[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:07 dist-test-slave-rzqm krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:07 dist-test-slave-rzqm krb5kdc[399](info): set up 2 sockets
Jun 20 04:30:07 dist-test-slave-rzqm krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 04:30:08.629101 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:08 dist-test-slave-rzqm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:08.749172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33727
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:08.859028 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:08.859331 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:08.859387 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:08.863514 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:08.863586 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:08.863620 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:08.863643 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:08.863669 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:08.869035 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:08.870110 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:08.871168 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:30:08.880213 412 server_base.cc:1034] running on GCE node
W20260620 04:30:08.880223 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:08.880200 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:08.880200 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:08.881192 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:08.882369 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:08.883564 412 hybrid_clock.cc:648] HybridClock initialized: now 1781929808883514 us; error 64 us; skew 500 ppm
Jun 20 04:30:08 dist-test-slave-rzqm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:08.886461 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:08.887328 412 webserver.cc:460] Webserver started at http://127.0.98.62:40567/ using document root <none> and password file <none>
I20260620 04:30:08.887866 412 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:08.887935 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:08.888135 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:08.890271 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "04325bbad592423a93d3682709e3380b"
format_stamp: "Formatted at 2026-06-20 04:30:08 on dist-test-slave-rzqm"
server_key: "99773ed7f38e1ac25223afd6bab567b9"
server_key_iv: "419d357f493cc617ac94bde720834a8d"
server_key_version: "encryptionkey@0"
I20260620 04:30:08.890801 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "04325bbad592423a93d3682709e3380b"
format_stamp: "Formatted at 2026-06-20 04:30:08 on dist-test-slave-rzqm"
server_key: "99773ed7f38e1ac25223afd6bab567b9"
server_key_iv: "419d357f493cc617ac94bde720834a8d"
server_key_version: "encryptionkey@0"
I20260620 04:30:08.894440 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:30:08.896936 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:08.898149 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:08.898259 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "04325bbad592423a93d3682709e3380b"
format_stamp: "Formatted at 2026-06-20 04:30:08 on dist-test-slave-rzqm"
server_key: "99773ed7f38e1ac25223afd6bab567b9"
server_key_iv: "419d357f493cc617ac94bde720834a8d"
server_key_version: "encryptionkey@0"
I20260620 04:30:08.898356 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:08.919118 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:08.922425 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:08.922629 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:08.929258 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33727
I20260620 04:30:08.929324 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33727 every 8 connection(s)
I20260620 04:30:08.930217 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:08.933108 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:08.935851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 412
I20260620 04:30:08.936033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:08.937724 392 external_mini_cluster.cc:1428] Setting key b35d14fdd9a430e8780985fc909f4d93
I20260620 04:30:08.939888 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b: Bootstrap starting.
I20260620 04:30:08.942752 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:08.943692 480 log.cc:824] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:08.946094 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b: No bootstrap required, opened a new log
I20260620 04:30:08.950529 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04325bbad592423a93d3682709e3380b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33727 } }
I20260620 04:30:08.950836 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:08.950904 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04325bbad592423a93d3682709e3380b, State: Initialized, Role: FOLLOWER
I20260620 04:30:08.951352 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04325bbad592423a93d3682709e3380b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33727 } }
I20260620 04:30:08.951496 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:08.951561 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:08.951656 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:08.952791 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04325bbad592423a93d3682709e3380b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33727 } }
I20260620 04:30:08.953202 480 leader_election.cc:302] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04325bbad592423a93d3682709e3380b; no voters:
I20260620 04:30:08.953586 480 leader_election.cc:288] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:08.953696 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:08.953938 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [term 1 LEADER]: Becoming Leader. State: Replica: 04325bbad592423a93d3682709e3380b, State: Running, Role: LEADER
I20260620 04:30:08.954276 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04325bbad592423a93d3682709e3380b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33727 } }
I20260620 04:30:08.954905 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:08.955976 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04325bbad592423a93d3682709e3380b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04325bbad592423a93d3682709e3380b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33727 } } }
I20260620 04:30:08.956099 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:08.956368 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04325bbad592423a93d3682709e3380b. Latest consensus state: current_term: 1 leader_uuid: "04325bbad592423a93d3682709e3380b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04325bbad592423a93d3682709e3380b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33727 } } }
I20260620 04:30:08.956455 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:08.956732 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 04:30:08 dist-test-slave-rzqm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:08.960177 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:08.965314 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:08.941525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.0.98.62:33727)
0620 04:30:08.942030 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:08.942050 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:08.943114 (+ 1064us) server_negotiation.cc:407] Connection header received
0620 04:30:08.947101 (+ 3987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:08.947117 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:08.947447 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:08.947738 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:08.950946 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:08.952149 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:08.953913 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:08.954668 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:08.954843 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:08.959562 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:08.959568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:08.959581 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:08.959613 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:08.961384 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:08.961970 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:08.961976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:08.961980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:08.962096 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:08.962584 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:08.962588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:08.962590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:08.963115 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:08.963319 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:08.964048 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 04:30:08.964332 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":152,"threads_started":1}
I20260620 04:30:08.966768 494 catalog_manager.cc:1347] Generated new cluster ID: 93e23832271646759a41a09913c2c219
I20260620 04:30:08.967046 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:08.988519 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:08.989879 494 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:09.001559 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04325bbad592423a93d3682709e3380b: Generated new TSK 0
I20260620 04:30:09.002336 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:30:09.031312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33727
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:09.141343 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:09.141634 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:09.141700 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:09.145709 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:09.145778 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:09.145862 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:09.151180 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:09.152277 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:09.153340 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:09.163187 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:09.163141 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:09.163193 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:09.163405 506 server_base.cc:1034] running on GCE node
I20260620 04:30:09.163872 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:09.164482 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:09.165652 506 hybrid_clock.cc:648] HybridClock initialized: now 1781929809165635 us; error 27 us; skew 500 ppm
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:09.168447 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:30:09.169404 506 webserver.cc:460] Webserver started at http://127.0.98.1:42889/ using document root <none> and password file <none>
I20260620 04:30:09.169945 506 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:09.170003 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:09.170176 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:09.172201 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "19cebb455f33453f8f0ffa3ace505013"
format_stamp: "Formatted at 2026-06-20 04:30:09 on dist-test-slave-rzqm"
server_key: "43c4fbea8bbf98e4fefe9da31ee384f0"
server_key_iv: "5b61f08ca76fc1b370eebf3bf7e60cf8"
server_key_version: "encryptionkey@0"
I20260620 04:30:09.172716 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "19cebb455f33453f8f0ffa3ace505013"
format_stamp: "Formatted at 2026-06-20 04:30:09 on dist-test-slave-rzqm"
server_key: "43c4fbea8bbf98e4fefe9da31ee384f0"
server_key_iv: "5b61f08ca76fc1b370eebf3bf7e60cf8"
server_key_version: "encryptionkey@0"
I20260620 04:30:09.176749 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:30:09.179332 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:09.180629 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:30:09.180750 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "19cebb455f33453f8f0ffa3ace505013"
format_stamp: "Formatted at 2026-06-20 04:30:09 on dist-test-slave-rzqm"
server_key: "43c4fbea8bbf98e4fefe9da31ee384f0"
server_key_iv: "5b61f08ca76fc1b370eebf3bf7e60cf8"
server_key_version: "encryptionkey@0"
I20260620 04:30:09.180855 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:09.210472 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:09.213828 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:09.214011 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:09.214617 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:09.215459 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:09.215508 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:09.215571 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:09.215595 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:09.223994 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38309
I20260620 04:30:09.224027 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38309 every 8 connection(s)
I20260620 04:30:09.224886 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:09.228045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 506
I20260620 04:30:09.228142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:09.228365 392 external_mini_cluster.cc:1428] Setting key 69eed1c0a195b2ced4d4b78934c9aeda
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:30:09.238930 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:09.226752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56475 (local address 127.0.98.62:33727)
0620 04:30:09.226880 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:09.226885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:09.227800 (+ 915us) server_negotiation.cc:407] Connection header received
0620 04:30:09.229213 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:09.229217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:09.229269 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:09.229341 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:09.230583 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.231102 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.232249 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.232472 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.232558 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:09.235250 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:09.235254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:09.235256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:09.235281 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:09.236800 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:09.237392 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:09.237395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:09.237396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:09.237484 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:09.237802 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:09.237805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:09.237806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:09.237987 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:09.238066 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:09.238621 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 04:30:09.238748 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:30:09.239724 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:09.227152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33727 (local address 127.0.98.1:56475)
0620 04:30:09.227630 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:09.227675 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:09.228953 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:09.229524 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:09.229534 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:09.229927 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:09.230430 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:09.230445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.231221 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.231224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:09.232132 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:09.232141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.232609 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.232613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:09.232778 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:09.233342 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:09.233359 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:09.235059 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:09.236934 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:09.236941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:09.236960 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:09.237297 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:09.237566 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:09.237568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:09.237569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:09.237725 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:09.238105 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:09.238110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:09.238357 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 04:30:09.238515 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 04:30:09.238790 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260620 04:30:09.241478 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33727
I20260620 04:30:09.241725 635 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:09.242215 635 heartbeater.cc:511] Master 127.0.98.62:33727 requested a full tablet report, sending...
I20260620 04:30:09.243773 444 ts_manager.cc:194] Registered new tserver with Master: 19cebb455f33453f8f0ffa3ace505013 (127.0.98.1:38309)
I20260620 04:30:09.244894 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:30:09.253904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33727
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:09.364563 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:09.364846 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:09.364908 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:09.368955 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:09.369045 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:09.369132 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:09.374470 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:09.375530 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:09.376574 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:09.386161 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:09.386281 640 server_base.cc:1034] running on GCE node
W20260620 04:30:09.386432 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:09.386344 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:09.386909 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:09.387543 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:09.388716 640 hybrid_clock.cc:648] HybridClock initialized: now 1781929809388697 us; error 31 us; skew 500 ppm
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:09.391475 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:30:09.392328 640 webserver.cc:460] Webserver started at http://127.0.98.2:36283/ using document root <none> and password file <none>
I20260620 04:30:09.392853 640 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:09.392917 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:09.393163 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:09.395234 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "4ab65b11e63645e6ac20e819be8f2481"
format_stamp: "Formatted at 2026-06-20 04:30:09 on dist-test-slave-rzqm"
server_key: "3d443cae692d32d050f66f5a36096af3"
server_key_iv: "747ce045f178a18fbc125e7b23310f46"
server_key_version: "encryptionkey@0"
I20260620 04:30:09.395797 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4ab65b11e63645e6ac20e819be8f2481"
format_stamp: "Formatted at 2026-06-20 04:30:09 on dist-test-slave-rzqm"
server_key: "3d443cae692d32d050f66f5a36096af3"
server_key_iv: "747ce045f178a18fbc125e7b23310f46"
server_key_version: "encryptionkey@0"
I20260620 04:30:09.399852 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 04:30:09.402917 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:09.404165 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:09.404277 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "4ab65b11e63645e6ac20e819be8f2481"
format_stamp: "Formatted at 2026-06-20 04:30:09 on dist-test-slave-rzqm"
server_key: "3d443cae692d32d050f66f5a36096af3"
server_key_iv: "747ce045f178a18fbc125e7b23310f46"
server_key_version: "encryptionkey@0"
I20260620 04:30:09.404377 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:09.426383 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:09.429422 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:09.429610 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:09.430215 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:09.431057 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:09.431104 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:09.431157 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:09.431185 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:09.440327 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43353
I20260620 04:30:09.440356 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43353 every 8 connection(s)
I20260620 04:30:09.441277 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:09.450358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 640
I20260620 04:30:09.450471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:09.450714 392 external_mini_cluster.cc:1428] Setting key 176e1684430718fa7adc45701c2340d9
I20260620 04:30:09.454849 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:09.443086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38927 (local address 127.0.98.62:33727)
0620 04:30:09.443228 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:09.443232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:09.444092 (+ 860us) server_negotiation.cc:407] Connection header received
0620 04:30:09.445505 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:09.445509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:09.445566 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:09.445649 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:09.446849 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.447366 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.448480 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.448701 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.448786 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:09.451231 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:09.451236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:09.451240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:09.451270 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:09.452683 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:09.453279 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:09.453282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:09.453283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:09.453361 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:09.453730 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:09.453734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:09.453736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:09.453954 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:09.454047 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:09.454570 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 04:30:09.454693 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:30:09.455700 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:09.443438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33727 (local address 127.0.98.2:38927)
0620 04:30:09.443929 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:09.443973 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:09.445300 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:09.445818 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:09.445825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:09.446188 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:09.446707 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:09.446721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.447480 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.447483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:09.448366 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:09.448375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.448837 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.448841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:09.448993 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:09.449563 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:09.449579 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:09.451054 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:09.452799 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:09.452806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:09.452823 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:09.453194 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:09.453435 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:09.453438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:09.453440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:09.453616 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:09.454044 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:09.454049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:09.454298 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 04:30:09.454451 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 04:30:09.454721 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":148,"threads_started":1}
I20260620 04:30:09.456817 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33727
I20260620 04:30:09.457103 769 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:09.457693 769 heartbeater.cc:511] Master 127.0.98.62:33727 requested a full tablet report, sending...
I20260620 04:30:09.458747 444 ts_manager.cc:194] Registered new tserver with Master: 4ab65b11e63645e6ac20e819be8f2481 (127.0.98.2:43353)
I20260620 04:30:09.459297 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38927
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:30:09.476676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33727
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:09.590023 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:09.590293 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:09.590338 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:09.594403 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:09.594467 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:09.594527 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:09.599879 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:09.600912 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:09.601989 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:09.612038 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:09.612211 774 server_base.cc:1034] running on GCE node
W20260620 04:30:09.612038 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:09.612125 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:09.612766 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:09.613435 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:09.614594 774 hybrid_clock.cc:648] HybridClock initialized: now 1781929809614569 us; error 40 us; skew 500 ppm
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:09.617281 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:30:09.618103 774 webserver.cc:460] Webserver started at http://127.0.98.3:46875/ using document root <none> and password file <none>
I20260620 04:30:09.618602 774 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:09.618655 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:09.618830 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:09.620805 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "ef43775b3bc24ed29c21eaf163d17978"
format_stamp: "Formatted at 2026-06-20 04:30:09 on dist-test-slave-rzqm"
server_key: "26ef988e3e74886db840a23175abe8af"
server_key_iv: "99b6e27a57542b36e8e9ada6d696fa45"
server_key_version: "encryptionkey@0"
I20260620 04:30:09.621338 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ef43775b3bc24ed29c21eaf163d17978"
format_stamp: "Formatted at 2026-06-20 04:30:09 on dist-test-slave-rzqm"
server_key: "26ef988e3e74886db840a23175abe8af"
server_key_iv: "99b6e27a57542b36e8e9ada6d696fa45"
server_key_version: "encryptionkey@0"
I20260620 04:30:09.624922 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:09.627285 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:09.628476 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:09.628568 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "ef43775b3bc24ed29c21eaf163d17978"
format_stamp: "Formatted at 2026-06-20 04:30:09 on dist-test-slave-rzqm"
server_key: "26ef988e3e74886db840a23175abe8af"
server_key_iv: "99b6e27a57542b36e8e9ada6d696fa45"
server_key_version: "encryptionkey@0"
I20260620 04:30:09.628649 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:09.637606 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:09.640558 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:09.640714 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:09.641264 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:09.642056 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:09.642100 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:09.642136 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:09.642156 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:09.650877 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44091
I20260620 04:30:09.650899 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44091 every 8 connection(s)
I20260620 04:30:09.651821 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:09.652848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 774
I20260620 04:30:09.652966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:09.653288 392 external_mini_cluster.cc:1428] Setting key 0cc5b2a4145ea247926a881b5f81c285
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:09.665764 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:09.653901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48111 (local address 127.0.98.62:33727)
0620 04:30:09.654051 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:09.654074 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:09.655104 (+ 1030us) server_negotiation.cc:407] Connection header received
0620 04:30:09.656557 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:09.656560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:09.656623 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:09.656692 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:09.657961 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.658485 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.659630 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.659842 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.659928 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:09.662450 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:09.662453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:09.662455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:09.662499 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:09.663884 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:09.664366 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:09.664369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:09.664370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:09.664460 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:09.664737 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:09.664739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:09.664740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:09.664921 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:09.664990 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:09.665498 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 04:30:09.665635 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:30:09.666623 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:09.654424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33727 (local address 127.0.98.3:48111)
0620 04:30:09.654971 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:09.655017 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:09.656354 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:09.656884 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:09.656892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:09.657322 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:09.657825 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:09.657839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.658599 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.658603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:09.659508 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:09.659517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.659980 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.659984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:09.660163 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:09.660736 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:09.660752 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:09.662265 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:09.663990 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:09.663995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:09.664008 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:09.664261 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:09.664534 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:09.664536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:09.664537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:09.664673 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:09.664994 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:09.665018 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:09.665282 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 04:30:09.665443 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 04:30:09.665731 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":163,"threads_started":1}
I20260620 04:30:09.667768 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33727
I20260620 04:30:09.668006 903 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:09.668495 903 heartbeater.cc:511] Master 127.0.98.62:33727 requested a full tablet report, sending...
I20260620 04:30:09.669504 444 ts_manager.cc:194] Registered new tserver with Master: ef43775b3bc24ed29c21eaf163d17978 (127.0.98.3:44091)
I20260620 04:30:09.670096 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48111
I20260620 04:30:09.679773 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:30:09.705431 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:09.695944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.0.98.1:38309)
0620 04:30:09.696274 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:09.696288 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:09.696314 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:30:09.696403 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:09.696409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:09.696599 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:09.696728 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:09.697476 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.698249 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.699072 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.699430 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.699507 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:09.701793 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:09.701803 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:09.701816 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:09.701841 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:09.703476 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:09.703967 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:09.703973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:09.703975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:09.704062 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:09.704329 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:09.704332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:09.704334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:09.704621 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:09.704832 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:09.705075 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 04:30:09.705240 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
W20260620 04:30:09.706446 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53806
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:30:09.731262 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:09.723148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.0.98.1:38309)
0620 04:30:09.723299 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:09.723304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:09.723320 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:30:09.723476 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:09.723480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:09.723523 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:09.723591 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:09.724274 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.724812 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.725708 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:09.725997 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:09.726098 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:09.728364 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:09.728368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:09.728370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:09.728396 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:09.729731 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:09.730145 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:09.730149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:09.730150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:09.730229 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:09.730474 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:09.730477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:09.730479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:09.730647 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:09.730757 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:09.730975 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 04:30:09.731102 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 04:30:09.732299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 506
I20260620 04:30:09.737387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 640
I20260620 04:30:09.744146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 774
I20260620 04:30:09.749102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 412
2026-06-20T04:30:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2086 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[917](info): set up 2 sockets
Jun 20 04:30:09 dist-test-slave-rzqm krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 04:30:10.708554 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:10 dist-test-slave-rzqm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:10.823499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:10.934381 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:10.934684 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:10.934726 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:10.939008 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:10.939083 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:10.939102 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:10.939118 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:10.939132 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:10.944758 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:10.945899 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:10.946934 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:10.955631 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:10.955639 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:10.955713 930 server_base.cc:1034] running on GCE node
W20260620 04:30:10.955648 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:10.956488 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:10.957676 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:10.958843 930 hybrid_clock.cc:648] HybridClock initialized: now 1781929810958822 us; error 31 us; skew 500 ppm
Jun 20 04:30:10 dist-test-slave-rzqm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:10.961686 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:10.962652 930 webserver.cc:460] Webserver started at http://127.0.98.62:36577/ using document root <none> and password file <none>
I20260620 04:30:10.963330 930 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:10.963404 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:10.963606 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:10.965759 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "51fb670770824a399af90d8d601e10b3"
format_stamp: "Formatted at 2026-06-20 04:30:10 on dist-test-slave-rzqm"
server_key: "d9115debd8e03d6ff420650369653b05"
server_key_iv: "dd1176de127dcb95db0bfc867699e1b9"
server_key_version: "encryptionkey@0"
I20260620 04:30:10.966303 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "51fb670770824a399af90d8d601e10b3"
format_stamp: "Formatted at 2026-06-20 04:30:10 on dist-test-slave-rzqm"
server_key: "d9115debd8e03d6ff420650369653b05"
server_key_iv: "dd1176de127dcb95db0bfc867699e1b9"
server_key_version: "encryptionkey@0"
I20260620 04:30:10.969936 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:30:10.972404 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:10.973685 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:10.973804 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "51fb670770824a399af90d8d601e10b3"
format_stamp: "Formatted at 2026-06-20 04:30:10 on dist-test-slave-rzqm"
server_key: "d9115debd8e03d6ff420650369653b05"
server_key_iv: "dd1176de127dcb95db0bfc867699e1b9"
server_key_version: "encryptionkey@0"
I20260620 04:30:10.973917 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:11.004674 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:11.008018 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:11.008231 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:11.014876 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42503
I20260620 04:30:11.014899 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42503 every 8 connection(s)
I20260620 04:30:11.015820 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:11.018608 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:11.019987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 930
I20260620 04:30:11.020089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:11.020330 392 external_mini_cluster.cc:1428] Setting key f33b77c1f2ca1745de0a4f29434f112f
I20260620 04:30:11.025266 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3: Bootstrap starting.
I20260620 04:30:11.027838 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:11.028663 998 log.cc:824] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:11.030721 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3: No bootstrap required, opened a new log
I20260620 04:30:11.034687 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fb670770824a399af90d8d601e10b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 04:30:11.034929 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:11.034984 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fb670770824a399af90d8d601e10b3, State: Initialized, Role: FOLLOWER
I20260620 04:30:11.035250 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.021834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.0.98.62:42503)
0620 04:30:11.022260 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.022281 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.022327 (+ 46us) server_negotiation.cc:407] Connection header received
0620 04:30:11.023165 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.023191 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.023637 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.024027 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:11.024670 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.026118 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.027031 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.027696 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.027864 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.030422 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:11.030427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:11.030441 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:11.030470 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:11.032224 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.032746 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.032750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.032755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.032875 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.033191 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.033194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.033195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.033601 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:11.033782 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.034047 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.034350 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":109,"threads_started":1}
I20260620 04:30:11.035449 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fb670770824a399af90d8d601e10b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 04:30:11.035585 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:11.035647 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:11.035744 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:11.036665 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fb670770824a399af90d8d601e10b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 04:30:11.037036 998 leader_election.cc:302] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51fb670770824a399af90d8d601e10b3; no voters:
I20260620 04:30:11.037387 998 leader_election.cc:288] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:11.037478 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:11.037649 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [term 1 LEADER]: Becoming Leader. State: Replica: 51fb670770824a399af90d8d601e10b3, State: Running, Role: LEADER
I20260620 04:30:11.038009 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fb670770824a399af90d8d601e10b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 04:30:11.038677 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:11.039748 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51fb670770824a399af90d8d601e10b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fb670770824a399af90d8d601e10b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } } }
I20260620 04:30:11.039881 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:11.040283 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:11.041154 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51fb670770824a399af90d8d601e10b3. Latest consensus state: current_term: 1 leader_uuid: "51fb670770824a399af90d8d601e10b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fb670770824a399af90d8d601e10b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } } }
I20260620 04:30:11.041280 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:11.043479 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:11.048976 1012 catalog_manager.cc:1347] Generated new cluster ID: 657c8cd2797b4283b76c8f5b1eeacd76
I20260620 04:30:11.049120 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:11.065196 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:11.066247 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:11.075683 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51fb670770824a399af90d8d601e10b3: Generated new TSK 0
I20260620 04:30:11.076496 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:30:11.103737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:11.241020 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:11.241294 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:11.241346 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:11.245415 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:11.245486 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:11.245546 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:11.250849 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:11.251888 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:11.252884 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:11.262688 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:11.262809 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:11.262800 1024 server_base.cc:1034] running on GCE node
W20260620 04:30:11.262825 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:11.263408 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:11.264055 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:11.265208 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781929811265190 us; error 33 us; skew 500 ppm
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:11.273731 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:30:11.274888 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34911/ using document root <none> and password file <none>
I20260620 04:30:11.275508 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:11.275584 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:11.275790 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:11.278038 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "e4171dd87e2544178772c6d87d7af389"
format_stamp: "Formatted at 2026-06-20 04:30:11 on dist-test-slave-rzqm"
server_key: "977e2f644e6054e4e6dbf57c7e84066d"
server_key_iv: "399808ab7839486c07069a2bedace66b"
server_key_version: "encryptionkey@0"
I20260620 04:30:11.278626 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e4171dd87e2544178772c6d87d7af389"
format_stamp: "Formatted at 2026-06-20 04:30:11 on dist-test-slave-rzqm"
server_key: "977e2f644e6054e4e6dbf57c7e84066d"
server_key_iv: "399808ab7839486c07069a2bedace66b"
server_key_version: "encryptionkey@0"
I20260620 04:30:11.282516 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:11.285218 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:11.286571 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:11.286691 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "e4171dd87e2544178772c6d87d7af389"
format_stamp: "Formatted at 2026-06-20 04:30:11 on dist-test-slave-rzqm"
server_key: "977e2f644e6054e4e6dbf57c7e84066d"
server_key_iv: "399808ab7839486c07069a2bedace66b"
server_key_version: "encryptionkey@0"
I20260620 04:30:11.286799 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:11.304229 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:11.307479 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:11.307662 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:11.308255 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:11.309124 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:11.309170 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:11.309232 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:11.309257 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:11.317755 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35035
I20260620 04:30:11.317827 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35035 every 8 connection(s)
I20260620 04:30:11.318673 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:11.320971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 1024
I20260620 04:30:11.321099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:11.321399 392 external_mini_cluster.cc:1428] Setting key bd54054e644a7ececcf1df5654ae2c47
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:30:11.334587 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.320546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60039 (local address 127.0.98.62:42503)
0620 04:30:11.320664 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.320667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.322143 (+ 1476us) server_negotiation.cc:407] Connection header received
0620 04:30:11.323645 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.323649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.323708 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.323789 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:11.325360 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.325931 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.327612 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.327853 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.327932 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.330888 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:11.330891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:11.330893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:11.330919 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:11.332295 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.332926 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.332930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.332931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.333038 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.333437 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.333440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.333441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.333626 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:11.333709 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.334267 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.334420 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:30:11.335423 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.320903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42503 (local address 127.0.98.1:60039)
0620 04:30:11.321992 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:11.322039 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:11.323421 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:11.323945 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:11.323952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:11.324454 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:11.325184 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.325205 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.326055 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.326058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.327481 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.327493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.328007 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.328011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.328248 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.328931 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:11.328953 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:11.330701 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:11.332440 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:11.332451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:11.332468 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:11.332773 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:11.333182 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:11.333185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:11.333186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:11.333334 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:11.333724 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:11.333731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:11.334026 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 04:30:11.334194 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 04:30:11.334539 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":133,"threads_started":1}
I20260620 04:30:11.336685 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42503
I20260620 04:30:11.336977 1153 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:11.337648 1153 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 04:30:11.339555 960 ts_manager.cc:194] Registered new tserver with Master: e4171dd87e2544178772c6d87d7af389 (127.0.98.1:35035)
I20260620 04:30:11.341406 960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:30:11.348301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:11.459332 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:11.459599 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:11.459642 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:11.463768 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:11.463833 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:11.463898 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:11.469233 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:11.470296 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:11.471315 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:11.481118 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:11.481285 1158 server_base.cc:1034] running on GCE node
W20260620 04:30:11.481173 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:11.481120 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:11.481846 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:11.482517 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:11.483683 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781929811483665 us; error 42 us; skew 500 ppm
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:11.486456 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:30:11.487345 1158 webserver.cc:460] Webserver started at http://127.0.98.2:46179/ using document root <none> and password file <none>
I20260620 04:30:11.487891 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:11.487957 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:11.488161 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:11.490327 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "87d3e8db33064b9c9c0035452a44394d"
format_stamp: "Formatted at 2026-06-20 04:30:11 on dist-test-slave-rzqm"
server_key: "f8e9ede4464f4694c590cb4966dbe4a6"
server_key_iv: "db2e9a28809ec89183b93477a5f1541e"
server_key_version: "encryptionkey@0"
I20260620 04:30:11.490942 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "87d3e8db33064b9c9c0035452a44394d"
format_stamp: "Formatted at 2026-06-20 04:30:11 on dist-test-slave-rzqm"
server_key: "f8e9ede4464f4694c590cb4966dbe4a6"
server_key_iv: "db2e9a28809ec89183b93477a5f1541e"
server_key_version: "encryptionkey@0"
I20260620 04:30:11.494859 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:30:11.497412 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:11.498736 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:11.498891 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "87d3e8db33064b9c9c0035452a44394d"
format_stamp: "Formatted at 2026-06-20 04:30:11 on dist-test-slave-rzqm"
server_key: "f8e9ede4464f4694c590cb4966dbe4a6"
server_key_iv: "db2e9a28809ec89183b93477a5f1541e"
server_key_version: "encryptionkey@0"
I20260620 04:30:11.498996 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:11.519232 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:11.523917 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:11.524127 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:11.525146 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:11.526425 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:11.526490 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:11.526556 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:11.526587 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:11.537739 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40443
I20260620 04:30:11.537753 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40443 every 8 connection(s)
I20260620 04:30:11.538949 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:11.544863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 1158
I20260620 04:30:11.544981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:11.545276 392 external_mini_cluster.cc:1428] Setting key d2c3c7ce6c656cbeefbae1634cf1ce8c
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:30:11.555341 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.541129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38319 (local address 127.0.98.62:42503)
0620 04:30:11.541262 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.541266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.542216 (+ 950us) server_negotiation.cc:407] Connection header received
0620 04:30:11.543621 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.543625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.543681 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.543766 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:11.545169 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.545729 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.547143 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.547375 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.547456 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.551343 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:11.551348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:11.551351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:11.551385 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:11.553065 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.553610 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.553614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.553616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.553716 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.554089 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.554093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.554094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.554330 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:11.554436 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.555059 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.555184 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:30:11.556275 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.541562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42503 (local address 127.0.98.2:38319)
0620 04:30:11.542049 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:11.542097 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:11.543407 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:11.543963 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:11.543970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:11.544345 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:11.544941 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.544963 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.545846 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.545849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.547012 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.547023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.547515 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.547519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.547715 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.549407 (+ 1692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:11.549424 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:11.551125 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:11.553186 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:11.553191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:11.553206 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:11.553485 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:11.553825 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:11.553828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:11.553829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:11.553982 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:11.554437 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:11.554445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:11.554751 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 04:30:11.554949 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 04:30:11.555268 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:30:11.557471 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42503
I20260620 04:30:11.557708 1287 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:11.558204 1287 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 04:30:11.559185 960 ts_manager.cc:194] Registered new tserver with Master: 87d3e8db33064b9c9c0035452a44394d (127.0.98.2:40443)
I20260620 04:30:11.559749 960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:30:11.570000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:11.682117 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:11.682418 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:11.682463 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:11.686652 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:11.686729 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:11.686794 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:11.692418 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:11.693610 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:11.694649 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:11.704547 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:11.704595 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:11.704595 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:11.704933 1292 server_base.cc:1034] running on GCE node
I20260620 04:30:11.705369 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:11.706065 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:11.707242 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781929811707202 us; error 46 us; skew 500 ppm
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:11.710140 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:30:11.711045 1292 webserver.cc:460] Webserver started at http://127.0.98.3:36499/ using document root <none> and password file <none>
I20260620 04:30:11.711582 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:11.711642 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:11.711827 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:11.713971 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "c4ec91a0b494402c882d55d532f0b134"
format_stamp: "Formatted at 2026-06-20 04:30:11 on dist-test-slave-rzqm"
server_key: "d1fc66e5f49a282ca5fa60d988a14f74"
server_key_iv: "09175f66f1bfb800556831f39c16205b"
server_key_version: "encryptionkey@0"
I20260620 04:30:11.714511 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c4ec91a0b494402c882d55d532f0b134"
format_stamp: "Formatted at 2026-06-20 04:30:11 on dist-test-slave-rzqm"
server_key: "d1fc66e5f49a282ca5fa60d988a14f74"
server_key_iv: "09175f66f1bfb800556831f39c16205b"
server_key_version: "encryptionkey@0"
I20260620 04:30:11.718255 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:11.720618 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:11.721942 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:11.722055 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "c4ec91a0b494402c882d55d532f0b134"
format_stamp: "Formatted at 2026-06-20 04:30:11 on dist-test-slave-rzqm"
server_key: "d1fc66e5f49a282ca5fa60d988a14f74"
server_key_iv: "09175f66f1bfb800556831f39c16205b"
server_key_version: "encryptionkey@0"
I20260620 04:30:11.722157 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:11.754330 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:11.757349 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:11.757524 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:11.758112 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:11.758934 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:11.758980 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:11.759033 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:11.759058 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:11.767408 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44383
I20260620 04:30:11.767434 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44383 every 8 connection(s)
I20260620 04:30:11.768303 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:11.776788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 1292
I20260620 04:30:11.776911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:11.777228 392 external_mini_cluster.cc:1428] Setting key fbd64ccfdeb002068fd04af3a28b655e
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:11.782577 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.770239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35363 (local address 127.0.98.62:42503)
0620 04:30:11.770404 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.770410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.771293 (+ 883us) server_negotiation.cc:407] Connection header received
0620 04:30:11.772598 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.772602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.772652 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.772730 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:11.774024 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.774574 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.775815 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.776071 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.776160 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.779119 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:11.779122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:11.779124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:11.779150 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:11.780539 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.781095 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.781098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.781099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.781175 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.781518 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.781521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.781523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.781736 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:11.781818 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.782342 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.782456 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 04:30:11.783373 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.770657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42503 (local address 127.0.98.3:35363)
0620 04:30:11.771130 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:11.771175 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:11.772371 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:11.772947 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:11.772954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:11.773368 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:11.773860 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.773884 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.774711 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.774715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.775692 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.775702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.776288 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.776292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.776461 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.777179 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:11.777200 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:11.778937 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:11.780652 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:11.780657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:11.780673 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:11.780959 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:11.781243 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:11.781245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:11.781247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:11.781390 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:11.781823 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:11.781829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:11.782084 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 04:30:11.782241 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 04:30:11.782541 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":105,"threads_started":1}
I20260620 04:30:11.784490 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42503
I20260620 04:30:11.784808 1421 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:11.785336 1421 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 04:30:11.786386 960 ts_manager.cc:194] Registered new tserver with Master: c4ec91a0b494402c882d55d532f0b134 (127.0.98.3:44383)
I20260620 04:30:11.786957 960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35363
I20260620 04:30:11.791330 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:30:11.801978 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.794785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.0.98.62:42503)
0620 04:30:11.794935 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.794940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.795163 (+ 223us) server_negotiation.cc:407] Connection header received
0620 04:30:11.795255 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.795258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.795301 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.795374 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:11.796042 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.796563 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.797500 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.797704 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.797772 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.799072 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:11.799075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:11.799077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:11.799099 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:11.800422 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.800791 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.800794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.800795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.800863 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.801199 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.801201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.801202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.801445 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:11.801547 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.801730 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.801824 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:30:11.807197 960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:11.810199 960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:11.824519 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.819390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35035 (local address 127.0.0.1:48994)
0620 04:30:11.819846 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:11.819857 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:11.819940 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:11.820597 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:11.820600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:11.820613 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:11.820683 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.820688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.822199 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.822202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.823106 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.823114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.823990 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.823994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.824094 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.824130 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 04:30:11.824232 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 04:30:11.824355 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":3,"thread_start_us":98,"threads_started":1}
I20260620 04:30:11.824681 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.818663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44383 (local address 127.0.0.1:40370)
0620 04:30:11.819468 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:11.819520 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:11.819658 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:11.820455 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:11.820461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:11.820499 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:11.820663 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.820671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.822123 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.822126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.823286 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.823295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.824278 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.824282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.824368 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.824389 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:30:11.824463 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 04:30:11.824537 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":50,"spinlock_wait_cycles":3328,"thread_start_us":91,"threads_started":1}
I20260620 04:30:11.824928 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.818943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.0.98.2:40443)
0620 04:30:11.819334 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.819345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.819817 (+ 472us) server_negotiation.cc:407] Connection header received
0620 04:30:11.819922 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.819927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.820123 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.820266 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:11.820773 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.821574 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.823046 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.823866 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.823973 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.824121 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.824587 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.824785 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":101,"threads_started":1}
I20260620 04:30:11.824975 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.818777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.0.98.3:44383)
0620 04:30:11.819467 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.819479 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.819797 (+ 318us) server_negotiation.cc:407] Connection header received
0620 04:30:11.819883 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.819889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.820120 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.820265 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:11.820785 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.821990 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.823410 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.824160 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.824262 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.824387 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.824497 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.824586 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":185,"threads_started":1}
I20260620 04:30:11.825171 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.819910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.0.98.1:35035)
0620 04:30:11.820198 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.820210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.820231 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:30:11.820296 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.820300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.820461 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.820812 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:11.820974 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.822078 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.823230 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.823871 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.823975 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.824121 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.824892 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.825021 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":97,"threads_started":1}
I20260620 04:30:11.824523 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.818664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40443 (local address 127.0.0.1:37826)
0620 04:30:11.819588 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:11.819601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:11.819719 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:11.820455 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:11.820461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:11.820495 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:11.820646 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.820654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.821690 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.821694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.822914 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.822923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.824193 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.824197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.824275 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.824300 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:30:11.824354 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 04:30:11.824419 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":73,"spinlock_wait_cycles":5760,"thread_start_us":100,"threads_started":1}
I20260620 04:30:11.827813 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 47ee29efb6c84c629d2a427c5ee1f1d0 (DEFAULT_TABLE table=test-table [id=1609eaf1a5ce44659e903a095a5b8416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:11.827848 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 47ee29efb6c84c629d2a427c5ee1f1d0 (DEFAULT_TABLE table=test-table [id=1609eaf1a5ce44659e903a095a5b8416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:11.827845 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 47ee29efb6c84c629d2a427c5ee1f1d0 (DEFAULT_TABLE table=test-table [id=1609eaf1a5ce44659e903a095a5b8416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:11.828745 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ee29efb6c84c629d2a427c5ee1f1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:11.828809 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ee29efb6c84c629d2a427c5ee1f1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:11.828809 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ee29efb6c84c629d2a427c5ee1f1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:11.835780 1440 tablet_bootstrap.cc:492] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134: Bootstrap starting.
I20260620 04:30:11.836524 1438 tablet_bootstrap.cc:492] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389: Bootstrap starting.
I20260620 04:30:11.837186 1439 tablet_bootstrap.cc:492] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d: Bootstrap starting.
I20260620 04:30:11.838936 1440 tablet_bootstrap.cc:654] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:11.839006 1438 tablet_bootstrap.cc:654] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:11.839126 1439 tablet_bootstrap.cc:654] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:11.840013 1440 log.cc:824] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:11.840489 1439 log.cc:824] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:11.840552 1438 log.cc:824] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:11.842165 1440 tablet_bootstrap.cc:492] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134: No bootstrap required, opened a new log
I20260620 04:30:11.842375 1438 tablet_bootstrap.cc:492] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389: No bootstrap required, opened a new log
I20260620 04:30:11.842391 1439 tablet_bootstrap.cc:492] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d: No bootstrap required, opened a new log
I20260620 04:30:11.842412 1440 ts_tablet_manager.cc:1397] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 04:30:11.842535 1438 ts_tablet_manager.cc:1397] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:30:11.842564 1439 ts_tablet_manager.cc:1397] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:30:11.845377 1439 raft_consensus.cc:348] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ec91a0b494402c882d55d532f0b134" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } } peers { permanent_uuid: "87d3e8db33064b9c9c0035452a44394d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40443 } } peers { permanent_uuid: "e4171dd87e2544178772c6d87d7af389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35035 } }
I20260620 04:30:11.845640 1439 raft_consensus.cc:374] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:11.845709 1439 raft_consensus.cc:729] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d3e8db33064b9c9c0035452a44394d, State: Initialized, Role: FOLLOWER
I20260620 04:30:11.846136 1439 consensus_queue.cc:260] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ec91a0b494402c882d55d532f0b134" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } } peers { permanent_uuid: "87d3e8db33064b9c9c0035452a44394d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40443 } } peers { permanent_uuid: "e4171dd87e2544178772c6d87d7af389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35035 } }
I20260620 04:30:11.846211 1440 raft_consensus.cc:348] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ec91a0b494402c882d55d532f0b134" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } } peers { permanent_uuid: "87d3e8db33064b9c9c0035452a44394d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40443 } } peers { permanent_uuid: "e4171dd87e2544178772c6d87d7af389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35035 } }
I20260620 04:30:11.846591 1440 raft_consensus.cc:374] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:11.846678 1440 raft_consensus.cc:729] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ec91a0b494402c882d55d532f0b134, State: Initialized, Role: FOLLOWER
I20260620 04:30:11.846808 1287 heartbeater.cc:503] Master 127.0.98.62:42503 was elected leader, sending a full tablet report...
I20260620 04:30:11.846572 1438 raft_consensus.cc:348] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ec91a0b494402c882d55d532f0b134" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } } peers { permanent_uuid: "87d3e8db33064b9c9c0035452a44394d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40443 } } peers { permanent_uuid: "e4171dd87e2544178772c6d87d7af389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35035 } }
I20260620 04:30:11.847183 1440 consensus_queue.cc:260] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ec91a0b494402c882d55d532f0b134" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } } peers { permanent_uuid: "87d3e8db33064b9c9c0035452a44394d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40443 } } peers { permanent_uuid: "e4171dd87e2544178772c6d87d7af389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35035 } }
I20260620 04:30:11.847259 1439 ts_tablet_manager.cc:1428] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:30:11.847365 1438 raft_consensus.cc:374] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:11.847461 1438 raft_consensus.cc:729] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4171dd87e2544178772c6d87d7af389, State: Initialized, Role: FOLLOWER
I20260620 04:30:11.847868 1421 heartbeater.cc:503] Master 127.0.98.62:42503 was elected leader, sending a full tablet report...
I20260620 04:30:11.847971 1438 consensus_queue.cc:260] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ec91a0b494402c882d55d532f0b134" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } } peers { permanent_uuid: "87d3e8db33064b9c9c0035452a44394d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40443 } } peers { permanent_uuid: "e4171dd87e2544178772c6d87d7af389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35035 } }
I20260620 04:30:11.848208 1440 ts_tablet_manager.cc:1428] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:30:11.848913 1438 ts_tablet_manager.cc:1428] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:30:11.849311 1153 heartbeater.cc:503] Master 127.0.98.62:42503 was elected leader, sending a full tablet report...
I20260620 04:30:11.894460 1446 raft_consensus.cc:482] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:11.894670 1446 raft_consensus.cc:504] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ec91a0b494402c882d55d532f0b134" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } } peers { permanent_uuid: "87d3e8db33064b9c9c0035452a44394d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40443 } } peers { permanent_uuid: "e4171dd87e2544178772c6d87d7af389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35035 } }
I20260620 04:30:11.895854 1446 leader_election.cc:288] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4ec91a0b494402c882d55d532f0b134 (127.0.98.3:44383), 87d3e8db33064b9c9c0035452a44394d (127.0.98.2:40443)
I20260620 04:30:11.900873 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.896208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47037 (local address 127.0.98.3:44383)
0620 04:30:11.896457 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.896462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.896712 (+ 250us) server_negotiation.cc:407] Connection header received
0620 04:30:11.896890 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.896893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.896938 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.897040 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:11.897281 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.898051 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.899661 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.900303 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.900419 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.900482 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.900676 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.900752 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260620 04:30:11.900897 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.896112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44383 (local address 127.0.98.1:47037)
0620 04:30:11.896625 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:11.896638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:11.896778 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:11.897040 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:11.897043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:11.897058 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:11.897187 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.897192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.898156 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.898159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.899539 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.899546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.900449 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.900452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.900548 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.900575 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 04:30:11.900650 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 04:30:11.900723 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":51,"thread_start_us":119,"threads_started":1}
I20260620 04:30:11.901106 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.896291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37001 (local address 127.0.98.2:40443)
0620 04:30:11.896432 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.896437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.896605 (+ 168us) server_negotiation.cc:407] Connection header received
0620 04:30:11.896747 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.896750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.896797 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.896862 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:11.897503 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.898209 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.899661 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.900303 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.900419 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.900482 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.900803 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.900971 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:30:11.901433 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.896114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40443 (local address 127.0.98.1:37001)
0620 04:30:11.896514 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:11.896527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:11.896636 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:11.897242 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:11.897244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:11.897262 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:11.897374 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.897381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.898296 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.898299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.899529 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:11.899540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.900562 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.900566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:11.900671 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.900694 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:30:11.901241 (+ 547us) client_negotiation.cc:240] Negotiation successful
0620 04:30:11.901319 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":29,"thread_start_us":84,"threads_started":1}
I20260620 04:30:11.901701 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ee29efb6c84c629d2a427c5ee1f1d0" candidate_uuid: "e4171dd87e2544178772c6d87d7af389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ec91a0b494402c882d55d532f0b134" is_pre_election: true
I20260620 04:30:11.902037 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ee29efb6c84c629d2a427c5ee1f1d0" candidate_uuid: "e4171dd87e2544178772c6d87d7af389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d3e8db33064b9c9c0035452a44394d" is_pre_election: true
I20260620 04:30:11.902062 1375 raft_consensus.cc:2454] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4171dd87e2544178772c6d87d7af389 in term 0.
I20260620 04:30:11.902315 1241 raft_consensus.cc:2454] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4171dd87e2544178772c6d87d7af389 in term 0.
I20260620 04:30:11.902575 1041 leader_election.cc:302] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4ec91a0b494402c882d55d532f0b134, e4171dd87e2544178772c6d87d7af389; no voters:
I20260620 04:30:11.902881 1446 raft_consensus.cc:2781] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:11.903009 1446 raft_consensus.cc:482] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:11.903095 1446 raft_consensus.cc:3037] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:11.904134 1446 raft_consensus.cc:504] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ec91a0b494402c882d55d532f0b134" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } } peers { permanent_uuid: "87d3e8db33064b9c9c0035452a44394d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40443 } } peers { permanent_uuid: "e4171dd87e2544178772c6d87d7af389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35035 } }
I20260620 04:30:11.904493 1446 leader_election.cc:288] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [CANDIDATE]: Term 1 election: Requested vote from peers c4ec91a0b494402c882d55d532f0b134 (127.0.98.3:44383), 87d3e8db33064b9c9c0035452a44394d (127.0.98.2:40443)
I20260620 04:30:11.904827 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ee29efb6c84c629d2a427c5ee1f1d0" candidate_uuid: "e4171dd87e2544178772c6d87d7af389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ec91a0b494402c882d55d532f0b134"
I20260620 04:30:11.904855 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ee29efb6c84c629d2a427c5ee1f1d0" candidate_uuid: "e4171dd87e2544178772c6d87d7af389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d3e8db33064b9c9c0035452a44394d"
I20260620 04:30:11.904937 1375 raft_consensus.cc:3037] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:11.904984 1241 raft_consensus.cc:3037] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:11.906064 1241 raft_consensus.cc:2454] T 47ee29efb6c84c629d2a427c5ee1f1d0 P 87d3e8db33064b9c9c0035452a44394d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4171dd87e2544178772c6d87d7af389 in term 1.
I20260620 04:30:11.906394 1043 leader_election.cc:302] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87d3e8db33064b9c9c0035452a44394d, e4171dd87e2544178772c6d87d7af389; no voters:
I20260620 04:30:11.906412 1375 raft_consensus.cc:2454] T 47ee29efb6c84c629d2a427c5ee1f1d0 P c4ec91a0b494402c882d55d532f0b134 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4171dd87e2544178772c6d87d7af389 in term 1.
I20260620 04:30:11.906551 1446 raft_consensus.cc:2781] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:11.906772 1446 raft_consensus.cc:686] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [term 1 LEADER]: Becoming Leader. State: Replica: e4171dd87e2544178772c6d87d7af389, State: Running, Role: LEADER
I20260620 04:30:11.907111 1446 consensus_queue.cc:237] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ec91a0b494402c882d55d532f0b134" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } } peers { permanent_uuid: "87d3e8db33064b9c9c0035452a44394d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40443 } } peers { permanent_uuid: "e4171dd87e2544178772c6d87d7af389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35035 } }
I20260620 04:30:11.910220 962 catalog_manager.cc:5495] T 47ee29efb6c84c629d2a427c5ee1f1d0 P e4171dd87e2544178772c6d87d7af389 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4171dd87e2544178772c6d87d7af389 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e4171dd87e2544178772c6d87d7af389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ec91a0b494402c882d55d532f0b134" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87d3e8db33064b9c9c0035452a44394d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4171dd87e2544178772c6d87d7af389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35035 } health_report { overall_health: HEALTHY } } }
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:30:11.927314 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.918081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.0.98.1:35035)
0620 04:30:11.918229 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.918234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.918337 (+ 103us) server_negotiation.cc:407] Connection header received
0620 04:30:11.918448 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.918450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.918494 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.918560 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:11.919173 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.919677 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.920511 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.920725 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.920805 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.923526 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:11.923541 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:11.923559 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:11.923588 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:11.925363 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.925841 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.925847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.925849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.925997 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.926288 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.926291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.926292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.926568 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:11.926795 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.927068 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.927188 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929811, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:30:11.953943 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.945695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49010 (local address 127.0.98.1:35035)
0620 04:30:11.945869 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.945873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.945971 (+ 98us) server_negotiation.cc:407] Connection header received
0620 04:30:11.946091 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.946093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.946136 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.946204 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:11.946892 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.947442 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.948249 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.948485 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.948580 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.950981 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:11.950984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:11.950986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:11.951014 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:11.952308 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.952700 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.952703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.952704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.952774 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.953122 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.953125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.953126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.953325 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:11.953436 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.953667 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.953776 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:30:11 dist-test-slave-rzqm krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:30:11.979996 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:11.971176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.0.98.1:35035)
0620 04:30:11.971297 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:11.971302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:11.971453 (+ 151us) server_negotiation.cc:407] Connection header received
0620 04:30:11.971592 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:11.971597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:11.971642 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:11.971709 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:11.972443 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.972946 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.973794 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:11.974009 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:11.974098 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:11.976666 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:11.976669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:11.976671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:11.976703 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:11.978176 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.978668 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.978671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.978672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.978754 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:11.979119 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:11.979122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:11.979123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:11.979306 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:11.979456 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:11.979706 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 04:30:11.979826 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 04:30:11.981367 1067 tablet_service.cc:2908] Rejecting scan request for tablet 47ee29efb6c84c629d2a427c5ee1f1d0: Uninitialized: clean time has not yet been initialized
I20260620 04:30:11.983453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 1024
I20260620 04:30:11.988739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 1158
I20260620 04:30:11.993307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 1292
I20260620 04:30:11.997807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 930
2026-06-20T04:30:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2248 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:12 dist-test-slave-rzqm krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:12 dist-test-slave-rzqm krb5kdc[1463](info): set up 2 sockets
Jun 20 04:30:12 dist-test-slave-rzqm krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260620 04:30:12.956522 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:13 dist-test-slave-rzqm krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:13.076646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38529
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:13.187172 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:13.187471 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:13.187532 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:13.191879 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:13.191969 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 04:30:13.192003 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:13.192028 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:13.192050 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:13.192080 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:13.197965 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38529
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:13.199117 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:13.200163 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:13.208818 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:13.208806 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:13.208884 1476 server_base.cc:1034] running on GCE node
W20260620 04:30:13.208810 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:13.209678 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:13.210783 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:13.211971 1476 hybrid_clock.cc:648] HybridClock initialized: now 1781929813211947 us; error 34 us; skew 500 ppm
Jun 20 04:30:13 dist-test-slave-rzqm krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:13.214846 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:13.215754 1476 webserver.cc:460] Webserver started at http://127.0.98.62:33423/ using document root <none> and password file <none>
I20260620 04:30:13.216291 1476 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:13.216356 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:13.216564 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:13.218619 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "2b213cf8533e4b2485b0070b381fca8f"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "cec0b73164a4c7d6341d96791d23fde9"
server_key_iv: "f3c8c2b99ef8aac8ce823c1883f9e35f"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.219166 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b213cf8533e4b2485b0070b381fca8f"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "cec0b73164a4c7d6341d96791d23fde9"
server_key_iv: "f3c8c2b99ef8aac8ce823c1883f9e35f"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.222826 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:30:13.225340 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:13.226727 1476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:13.226850 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "2b213cf8533e4b2485b0070b381fca8f"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "cec0b73164a4c7d6341d96791d23fde9"
server_key_iv: "f3c8c2b99ef8aac8ce823c1883f9e35f"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.226966 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:13.255330 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:13.258783 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:13.258967 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:13.266624 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38529
I20260620 04:30:13.266671 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38529 every 8 connection(s)
I20260620 04:30:13.267621 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:13.270606 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:13.273265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 1476
I20260620 04:30:13.273367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:13.273602 392 external_mini_cluster.cc:1428] Setting key e4ea9d1b4e8eedfc1e37bc533709d7c3
I20260620 04:30:13.276491 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f: Bootstrap starting.
I20260620 04:30:13.279035 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:13.279943 1554 log.cc:824] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:13.282222 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f: No bootstrap required, opened a new log
Jun 20 04:30:13 dist-test-slave-rzqm krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:13.286798 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b213cf8533e4b2485b0070b381fca8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38529 } }
I20260620 04:30:13.287122 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:13.287184 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b213cf8533e4b2485b0070b381fca8f, State: Initialized, Role: FOLLOWER
I20260620 04:30:13.287755 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b213cf8533e4b2485b0070b381fca8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38529 } }
I20260620 04:30:13.287937 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:13.288002 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:13.288092 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:13.288906 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.275102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.0.98.62:38529)
0620 04:30:13.275532 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:13.275547 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:13.275584 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:30:13.276230 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:13.276248 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:13.276594 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:13.276899 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:13.277816 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.278839 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.280007 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.280562 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.280696 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.283491 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:13.283498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:13.283516 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:13.283553 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:13.285718 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.286191 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.286198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.286205 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.286355 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.286608 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.286611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.286613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.287149 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:13.287363 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:13.287582 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 04:30:13.287882 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260620 04:30:13.289321 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b213cf8533e4b2485b0070b381fca8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38529 } }
I20260620 04:30:13.289687 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b213cf8533e4b2485b0070b381fca8f; no voters:
I20260620 04:30:13.290032 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:13.290134 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:13.290341 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [term 1 LEADER]: Becoming Leader. State: Replica: 2b213cf8533e4b2485b0070b381fca8f, State: Running, Role: LEADER
I20260620 04:30:13.290714 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b213cf8533e4b2485b0070b381fca8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38529 } }
I20260620 04:30:13.291390 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:13.292459 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b213cf8533e4b2485b0070b381fca8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b213cf8533e4b2485b0070b381fca8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38529 } } }
I20260620 04:30:13.292480 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b213cf8533e4b2485b0070b381fca8f. Latest consensus state: current_term: 1 leader_uuid: "2b213cf8533e4b2485b0070b381fca8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b213cf8533e4b2485b0070b381fca8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38529 } } }
I20260620 04:30:13.292613 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:13.292980 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:13.292613 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:13.296418 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:13.301903 1568 catalog_manager.cc:1347] Generated new cluster ID: 1cde61eb7f614aebb7c8cbf22aca7293
I20260620 04:30:13.301967 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:13.311328 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:13.312328 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:13.317772 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f: Generated new TSK 0
I20260620 04:30:13.318605 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:30:13.352104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38529
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:13.487236 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:13.487538 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:13.487583 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:13.491751 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:13.491847 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:13.491891 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:30:13.491925 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:13.497493 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:13.498617 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:13.499616 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:13.509684 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:13.509686 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:13.509728 1580 server_base.cc:1034] running on GCE node
W20260620 04:30:13.509690 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:13.510305 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:13.511029 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:13.512235 1580 hybrid_clock.cc:648] HybridClock initialized: now 1781929813512211 us; error 33 us; skew 500 ppm
Jun 20 04:30:13 dist-test-slave-rzqm krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:13.515326 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:30:13.516331 1580 webserver.cc:460] Webserver started at http://127.0.98.1:42595/ using document root <none> and password file <none>
I20260620 04:30:13.516889 1580 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:13.516952 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:13.517176 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:13.519361 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "b79d31e28880415e83c102e65623a2f4"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "8564274dc44fa34fd985b82d4bbded9d"
server_key_iv: "26e33e8d89e78e44f820868f70bcf9fb"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.519917 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b79d31e28880415e83c102e65623a2f4"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "8564274dc44fa34fd985b82d4bbded9d"
server_key_iv: "26e33e8d89e78e44f820868f70bcf9fb"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.523893 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:30:13.526464 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:13.527823 1580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:13.527947 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "b79d31e28880415e83c102e65623a2f4"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "8564274dc44fa34fd985b82d4bbded9d"
server_key_iv: "26e33e8d89e78e44f820868f70bcf9fb"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.528069 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:13.549873 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:13.553274 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:13.553478 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:13.554970 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:13.555025 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:13.555148 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:13.555178 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:13.565518 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35725
Jun 20 04:30:13 dist-test-slave-rzqm krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:13.565649 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35725 every 8 connection(s)
I20260620 04:30:13.566566 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:13.569664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 1580
I20260620 04:30:13.569783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:13.570066 392 external_mini_cluster.cc:1428] Setting key af4e0d67ee658965f3af92076197c7b7
I20260620 04:30:13.570117 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.556080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60487 (local address 127.0.98.62:38529)
0620 04:30:13.556206 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:13.556209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:13.557428 (+ 1219us) server_negotiation.cc:407] Connection header received
0620 04:30:13.558551 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:13.558555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:13.558615 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:13.558697 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:13.560405 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.561168 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.562972 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.563222 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.563310 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.566324 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:13.566328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:13.566331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:13.566362 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:13.567915 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.568466 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.568469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.568470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.568546 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.568925 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.568928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.568930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.569169 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:13.569256 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:13.569852 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 04:30:13.569980 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:30:13.571134 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.556658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38529 (local address 127.0.98.1:60487)
0620 04:30:13.557260 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:13.557314 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:13.558309 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:13.558966 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:13.558976 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:13.559505 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:13.560245 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.560267 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.561308 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.561321 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:13.562766 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.562778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.563383 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.563388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:13.563620 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.564318 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:13.564339 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:13.566127 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:13.568070 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:13.568080 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:13.568092 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:13.568362 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:13.568659 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:13.568662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:13.568663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:13.568813 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:13.569272 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:13.569278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:13.569559 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 04:30:13.569770 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 04:30:13.570178 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":141,"threads_started":1}
I20260620 04:30:13.572304 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38529
I20260620 04:30:13.572446 1711 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:13.572919 1711 heartbeater.cc:511] Master 127.0.98.62:38529 requested a full tablet report, sending...
I20260620 04:30:13.575079 1507 ts_manager.cc:194] Registered new tserver with Master: b79d31e28880415e83c102e65623a2f4 (127.0.98.1:35725)
I20260620 04:30:13.576577 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60487
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:30:13.583499 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.577210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32831 (local address 127.0.98.62:38529)
0620 04:30:13.577344 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:13.577349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:13.577363 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:13.577400 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:13.577403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:13.577446 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:13.577521 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:13.577973 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.578477 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.579247 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.579466 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.579549 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.580724 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:13.580727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:13.580729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:13.580751 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:13.581944 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.582316 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.582319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.582321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.582389 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.582627 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.582630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.582632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.582796 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:13.582880 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:13.583232 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 04:30:13.583345 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:30:13.583499 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.574536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38529 (local address 127.0.98.1:32831)
0620 04:30:13.576953 (+ 2417us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:13.576969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:13.577101 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:13.577562 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:13.577564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:13.577735 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:13.577865 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.577870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.578591 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.578594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:13.579147 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.579155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.579559 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.579563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:13.579644 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.580226 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:13.580243 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:13.580584 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:13.582046 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:13.582049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:13.582051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:13.582233 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:13.582470 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:13.582472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:13.582474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:13.582552 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:13.582884 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:13.582887 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:13.583044 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 04:30:13.583147 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 04:30:13.583293 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2321,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:30:13.598109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38529
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:13.708245 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:13.708519 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:13.708565 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:13.712627 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:13.712700 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:13.712741 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:30:13.712780 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:13.718308 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:13.719403 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:13.720428 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:13.729875 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:13.729875 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:13.729933 1720 server_base.cc:1034] running on GCE node
W20260620 04:30:13.729869 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:13.730517 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:13.731204 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:13.732371 1720 hybrid_clock.cc:648] HybridClock initialized: now 1781929813732343 us; error 41 us; skew 500 ppm
Jun 20 04:30:13 dist-test-slave-rzqm krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:13.735080 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:30:13.735925 1720 webserver.cc:460] Webserver started at http://127.0.98.2:42349/ using document root <none> and password file <none>
I20260620 04:30:13.736441 1720 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:13.736505 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:13.736718 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:13.738760 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "2b7901fbacc34b54a7fffc12ca02bc84"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "d199a86962df3bd998bccd9e179b5f24"
server_key_iv: "5fd5b98a77436fca4a68d475f93cbd96"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.739279 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2b7901fbacc34b54a7fffc12ca02bc84"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "d199a86962df3bd998bccd9e179b5f24"
server_key_iv: "5fd5b98a77436fca4a68d475f93cbd96"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.742915 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:30:13.745324 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:13.746567 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:30:13.746693 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "2b7901fbacc34b54a7fffc12ca02bc84"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "d199a86962df3bd998bccd9e179b5f24"
server_key_iv: "5fd5b98a77436fca4a68d475f93cbd96"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.746789 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:13.758757 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:13.761886 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:13.762073 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:13.763643 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:13.763710 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:13.763768 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:13.763794 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:30:13 dist-test-slave-rzqm krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:13.773840 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46673
I20260620 04:30:13.773890 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46673 every 8 connection(s)
I20260620 04:30:13.774932 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:13.777396 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.764477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54431 (local address 127.0.98.62:38529)
0620 04:30:13.764627 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:13.764632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:13.765668 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 04:30:13.766820 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:13.766824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:13.766886 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:13.766969 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:13.768721 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.769480 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.770737 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.770959 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.771050 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.773932 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:13.773936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:13.773939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:13.773972 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:13.775327 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.775905 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.775909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.775911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.776012 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.776397 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.776399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.776400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.776578 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:13.776655 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:13.777167 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 04:30:13.777276 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:30:13.778267 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.764960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38529 (local address 127.0.98.2:54431)
0620 04:30:13.765480 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:13.765534 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:13.766592 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:13.767223 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:13.767233 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:13.767751 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:13.768525 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.768548 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.769615 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.769619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:13.770608 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.770618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.771109 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.771113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:13.771284 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.771920 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:13.771939 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:13.773750 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:13.775463 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:13.775468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:13.775484 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:13.775800 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:13.776167 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:13.776170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:13.776171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:13.776315 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:13.776658 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:13.776663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:13.776901 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 04:30:13.777083 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 04:30:13.777373 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":124,"threads_started":1}
I20260620 04:30:13.779467 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38529
I20260620 04:30:13.779683 1851 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:13.780259 1851 heartbeater.cc:511] Master 127.0.98.62:38529 requested a full tablet report, sending...
I20260620 04:30:13.781458 1508 ts_manager.cc:194] Registered new tserver with Master: 2b7901fbacc34b54a7fffc12ca02bc84 (127.0.98.2:46673)
I20260620 04:30:13.782199 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54431
I20260620 04:30:13.784628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 1720
I20260620 04:30:13.784727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:13.785018 392 external_mini_cluster.cc:1428] Setting key fbb3824348f511f3b296e7b43db1750e
I20260620 04:30:13.789248 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.781936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50769 (local address 127.0.98.62:38529)
0620 04:30:13.782049 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:13.782053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:13.782289 (+ 236us) server_negotiation.cc:407] Connection header received
0620 04:30:13.782409 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:13.782411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:13.782458 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:13.782525 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:13.782998 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.783728 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.784639 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.784905 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.784978 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.786213 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:13.786217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:13.786219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:13.786243 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:13.787545 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.788039 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.788042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.788044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.788138 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.788410 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.788412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.788414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.788678 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:13.788772 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:13.788971 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 04:30:13.789097 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:30:13.789256 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.781839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38529 (local address 127.0.98.2:50769)
0620 04:30:13.782199 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:13.782214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:13.782317 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:13.782561 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:13.782564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:13.782781 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:13.782886 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.782893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.783869 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.783872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:13.784510 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.784520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.785052 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.785057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:13.785147 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.785664 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:13.785678 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:13.786058 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:13.787681 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:13.787685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:13.787688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:13.787944 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:13.788224 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:13.788227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:13.788229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:13.788327 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:13.788763 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:13.788765 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:13.788844 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 04:30:13.788922 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 04:30:13.789090 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:30:13.809746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38529
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:13.920631 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:13.920902 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:13.920946 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:13.925053 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:13.925122 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:13.925161 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:30:13.925194 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:13.930724 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:13.931869 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:13.932912 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:30:13.942549 1860 server_base.cc:1034] running on GCE node
W20260620 04:30:13.942493 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:13.942492 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:13.942493 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:13.943260 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:13.943892 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:13.945089 1860 hybrid_clock.cc:648] HybridClock initialized: now 1781929813945070 us; error 36 us; skew 500 ppm
Jun 20 04:30:13 dist-test-slave-rzqm krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:13.947870 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:30:13.948822 1860 webserver.cc:460] Webserver started at http://127.0.98.3:39285/ using document root <none> and password file <none>
I20260620 04:30:13.949456 1860 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:13.949535 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:13.949766 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:13.951877 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "62779dda1c004eaa90a9a73a5b8fda0e"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "32952b8d2ac29bfd74f30142c384f2fc"
server_key_iv: "0d1e4c0a830c896d2ba32c16dc1f1905"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.952440 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "62779dda1c004eaa90a9a73a5b8fda0e"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "32952b8d2ac29bfd74f30142c384f2fc"
server_key_iv: "0d1e4c0a830c896d2ba32c16dc1f1905"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.956324 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:30:13.958995 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:13.960268 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:30:13.960392 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "62779dda1c004eaa90a9a73a5b8fda0e"
format_stamp: "Formatted at 2026-06-20 04:30:13 on dist-test-slave-rzqm"
server_key: "32952b8d2ac29bfd74f30142c384f2fc"
server_key_iv: "0d1e4c0a830c896d2ba32c16dc1f1905"
server_key_version: "encryptionkey@0"
I20260620 04:30:13.960496 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:13.974428 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:13.978050 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:13.978267 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:13.979789 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:13.979842 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:13.979894 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:13.979945 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:30:13 dist-test-slave-rzqm krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:13.989920 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41053
I20260620 04:30:13.990053 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41053 every 8 connection(s)
I20260620 04:30:13.990900 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:13.993115 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.980869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47349 (local address 127.0.98.62:38529)
0620 04:30:13.981092 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:13.981096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:13.982125 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 04:30:13.983236 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:13.983239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:13.983299 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:13.983370 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:13.984686 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.985272 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.986608 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.986892 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.986994 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.989477 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:13.989487 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:13.989489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:13.989515 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:13.990948 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.991537 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.991542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.991543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.991645 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:13.991977 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:13.991979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:13.991981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:13.992219 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:13.992306 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:13.992806 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 04:30:13.992955 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:30:13.993917 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.981467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38529 (local address 127.0.98.3:47349)
0620 04:30:13.981949 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:13.981998 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:13.983018 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:13.983574 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:13.983581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:13.983932 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:13.984533 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.984549 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.985391 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.985419 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:13.986490 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.986500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.987058 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.987062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:13.987223 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:13.987770 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:13.987786 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:13.989307 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:13.991049 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:13.991055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:13.991075 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:13.991411 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:13.991724 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:13.991727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:13.991729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:13.991876 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:13.992328 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:13.992334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:13.992579 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 04:30:13.992737 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 04:30:13.993054 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":134,"threads_started":1}
I20260620 04:30:13.995039 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38529
I20260620 04:30:13.995188 1991 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:13.995765 1991 heartbeater.cc:511] Master 127.0.98.62:38529 requested a full tablet report, sending...
I20260620 04:30:13.995832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 1860
I20260620 04:30:13.995962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:13.996237 392 external_mini_cluster.cc:1428] Setting key 18bf01a700e8b1d75ed92b68e9aed8d6
I20260620 04:30:13.996948 1507 ts_manager.cc:194] Registered new tserver with Master: 62779dda1c004eaa90a9a73a5b8fda0e (127.0.98.3:41053)
I20260620 04:30:13.997678 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47349
I20260620 04:30:13.998224 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:30:14.004633 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.997590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36063 (local address 127.0.98.62:38529)
0620 04:30:13.997711 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:13.997715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:13.998238 (+ 523us) server_negotiation.cc:407] Connection header received
0620 04:30:13.998363 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:13.998367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:13.998412 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:13.998489 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:13.999025 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.999752 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.000567 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.000804 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.000888 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.001937 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:14.001941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:14.001944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:14.001967 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:14.003221 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:14.003583 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:14.003587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:14.003588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:14.003667 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:14.003895 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:14.003898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:14.003899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:14.004067 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:14.004149 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.004385 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.004482 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:30:14.004644 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:13.997348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38529 (local address 127.0.98.3:36063)
0620 04:30:13.998154 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:13.998169 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:13.998280 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:13.998519 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:13.998522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:13.998760 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:13.998911 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:13.998920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:13.999866 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:13.999870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.000465 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.000473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.000898 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.000902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.000990 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.001484 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:14.001498 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:14.001815 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:14.003317 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:14.003320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:14.003322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:14.003503 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:14.003743 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:14.003746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:14.003748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:14.003822 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:14.004159 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:14.004162 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:14.004245 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.004326 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.004458 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
Jun 20 04:30:14 dist-test-slave-rzqm krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929814, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:30:14 dist-test-slave-rzqm krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929814, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:14.031601 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.022479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.0.98.62:38529)
0620 04:30:14.022636 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.022642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.022661 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:30:14.022714 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.022717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.022773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.022882 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:14.023958 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.024502 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.025484 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.025755 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.025861 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.028407 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:14.028410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:14.028412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:14.028439 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:14.029826 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:14.030336 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:14.030339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:14.030341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:14.030455 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:14.030753 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:14.030757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:14.030759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:14.030949 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:14.031077 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.031306 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.031427 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:30:14.034863 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:14.038048 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:14.052605 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.046752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.0.98.2:46673)
0620 04:30:14.048096 (+ 1344us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.048107 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.048127 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:30:14.048195 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.048200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.048375 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.048783 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:14.048915 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.050013 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.051120 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.051952 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.052059 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.052229 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.052375 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.052457 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":117,"threads_started":1}
I20260620 04:30:14.052627 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.046631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46673 (local address 127.0.0.1:37072)
0620 04:30:14.046959 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:14.047008 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:14.047141 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:14.048510 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:14.048513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:14.048539 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:14.048638 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.048643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.050110 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.050114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.051001 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.051011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.052065 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.052069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.052182 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.052224 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.052344 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.052465 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":21,"thread_start_us":78,"threads_started":1}
I20260620 04:30:14.053105 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.046818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35725 (local address 127.0.0.1:36352)
0620 04:30:14.047364 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:14.047376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:14.047455 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:14.048405 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:14.048411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:14.048432 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:14.048592 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.048599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.050001 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.050004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.051676 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.051684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.052600 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.052604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.052697 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.052721 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.052803 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.052892 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260620 04:30:14.053117 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.047300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.0.98.1:35725)
0620 04:30:14.047810 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.047825 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.047847 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:30:14.047921 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.047927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.048215 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.048408 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:14.048708 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.049897 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.051791 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.052484 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.052582 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.052735 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.052837 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.052931 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":75,"threads_started":1}
I20260620 04:30:14.053409 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.047597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.0.98.3:41053)
0620 04:30:14.047951 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.047962 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.047983 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:30:14.048064 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.048073 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.048284 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.048430 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:14.048735 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.049583 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.051174 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.051950 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.052064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.052229 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.053180 (+ 951us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.053284 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":94,"threads_started":1}
I20260620 04:30:14.053429 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.047345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41053 (local address 127.0.0.1:35458)
0620 04:30:14.047583 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:14.047596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:14.047683 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:14.048555 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:14.048557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:14.048571 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:14.048652 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.048658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.049680 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.049683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.051060 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.051069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.052792 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.052796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.052905 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.052927 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.052995 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.053104 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":64,"thread_start_us":94,"threads_started":1}
I20260620 04:30:14.056105 1925 tablet_service.cc:1467] Processing CreateTablet for tablet a6761637f8644e2e9147ed9fcf57dc97 (DEFAULT_TABLE table=test-table [id=46fb7974c11944528011e32f8b3bded0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:14.056174 1785 tablet_service.cc:1467] Processing CreateTablet for tablet a6761637f8644e2e9147ed9fcf57dc97 (DEFAULT_TABLE table=test-table [id=46fb7974c11944528011e32f8b3bded0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:14.056175 1645 tablet_service.cc:1467] Processing CreateTablet for tablet a6761637f8644e2e9147ed9fcf57dc97 (DEFAULT_TABLE table=test-table [id=46fb7974c11944528011e32f8b3bded0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:14.057145 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6761637f8644e2e9147ed9fcf57dc97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:14.057204 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6761637f8644e2e9147ed9fcf57dc97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:14.057204 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6761637f8644e2e9147ed9fcf57dc97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:14.063737 2014 tablet_bootstrap.cc:492] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: Bootstrap starting.
I20260620 04:30:14.066732 2014 tablet_bootstrap.cc:654] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:14.067462 2014 log.cc:824] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:14.068151 2015 tablet_bootstrap.cc:492] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: Bootstrap starting.
I20260620 04:30:14.068652 2016 tablet_bootstrap.cc:492] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: Bootstrap starting.
I20260620 04:30:14.069610 2014 tablet_bootstrap.cc:492] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: No bootstrap required, opened a new log
I20260620 04:30:14.069806 2014 ts_tablet_manager.cc:1397] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 04:30:14.070405 2016 tablet_bootstrap.cc:654] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:14.070417 2015 tablet_bootstrap.cc:654] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:14.071210 2016 log.cc:824] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:14.071596 2015 log.cc:824] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:14.073293 2016 tablet_bootstrap.cc:492] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: No bootstrap required, opened a new log
I20260620 04:30:14.073474 2016 ts_tablet_manager.cc:1397] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:30:14.073565 2015 tablet_bootstrap.cc:492] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: No bootstrap required, opened a new log
I20260620 04:30:14.073745 2015 ts_tablet_manager.cc:1397] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:30:14.073611 2014 raft_consensus.cc:348] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } }
I20260620 04:30:14.073872 2014 raft_consensus.cc:374] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:14.073975 2014 raft_consensus.cc:729] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62779dda1c004eaa90a9a73a5b8fda0e, State: Initialized, Role: FOLLOWER
I20260620 04:30:14.074435 2014 consensus_queue.cc:260] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } }
I20260620 04:30:14.075110 1991 heartbeater.cc:503] Master 127.0.98.62:38529 was elected leader, sending a full tablet report...
I20260620 04:30:14.075433 2014 ts_tablet_manager.cc:1428] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:30:14.076682 2016 raft_consensus.cc:348] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } }
I20260620 04:30:14.076892 2016 raft_consensus.cc:374] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:14.076962 2016 raft_consensus.cc:729] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7901fbacc34b54a7fffc12ca02bc84, State: Initialized, Role: FOLLOWER
I20260620 04:30:14.077399 2016 consensus_queue.cc:260] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } }
I20260620 04:30:14.078387 2016 ts_tablet_manager.cc:1428] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 04:30:14.078780 1851 heartbeater.cc:503] Master 127.0.98.62:38529 was elected leader, sending a full tablet report...
I20260620 04:30:14.078480 2015 raft_consensus.cc:348] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } }
I20260620 04:30:14.079764 2015 raft_consensus.cc:374] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:14.079829 2015 raft_consensus.cc:729] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79d31e28880415e83c102e65623a2f4, State: Initialized, Role: FOLLOWER
I20260620 04:30:14.080281 2015 consensus_queue.cc:260] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } }
I20260620 04:30:14.080924 1711 heartbeater.cc:503] Master 127.0.98.62:38529 was elected leader, sending a full tablet report...
I20260620 04:30:14.081249 2015 ts_tablet_manager.cc:1428] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:30:14.090710 2021 raft_consensus.cc:482] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:14.090904 2021 raft_consensus.cc:504] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } }
I20260620 04:30:14.092089 2021 leader_election.cc:288] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62779dda1c004eaa90a9a73a5b8fda0e (127.0.98.3:41053), b79d31e28880415e83c102e65623a2f4 (127.0.98.1:35725)
I20260620 04:30:14.095677 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.092362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35725 (local address 127.0.98.2:48339)
0620 04:30:14.092472 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:14.092487 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:14.092613 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:14.092879 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:14.092882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:14.092901 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:14.093041 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.093048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.093766 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.093770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.094586 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.094595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.095258 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.095262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.095365 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.095390 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.095470 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.095550 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 04:30:14.096433 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.092454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48339 (local address 127.0.98.1:35725)
0620 04:30:14.092580 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.092584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.092599 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:14.092722 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.092725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.092766 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.092865 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:14.093152 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.093663 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.094708 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.095144 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.095241 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.095298 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.096256 (+ 958us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.096340 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:30:14.096598 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.092454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56383 (local address 127.0.98.3:41053)
0620 04:30:14.092617 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.092622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.092971 (+ 349us) server_negotiation.cc:407] Connection header received
0620 04:30:14.093104 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.093107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.093195 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.093263 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:14.093848 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.094383 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.095508 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.095988 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.096144 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.096205 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.096389 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.096471 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 04:30:14.097038 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6761637f8644e2e9147ed9fcf57dc97" candidate_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79d31e28880415e83c102e65623a2f4" is_pre_election: true
I20260620 04:30:14.097385 1665 raft_consensus.cc:2454] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7901fbacc34b54a7fffc12ca02bc84 in term 0.
I20260620 04:30:14.097718 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.092362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41053 (local address 127.0.98.2:56383)
0620 04:30:14.092877 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:14.092888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:14.092994 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:14.093272 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:14.093274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:14.093289 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:14.093410 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.093416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.094487 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.094491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.095383 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.095391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.096184 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.096187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.096261 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.096280 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.096348 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.096418 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":163,"thread_start_us":124,"threads_started":1}
I20260620 04:30:14.097900 1738 leader_election.cc:302] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7901fbacc34b54a7fffc12ca02bc84, b79d31e28880415e83c102e65623a2f4; no voters:
I20260620 04:30:14.098193 2021 raft_consensus.cc:2781] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:14.098277 2021 raft_consensus.cc:482] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:14.098327 2021 raft_consensus.cc:3037] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:14.098320 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6761637f8644e2e9147ed9fcf57dc97" candidate_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" is_pre_election: true
I20260620 04:30:14.098596 1945 raft_consensus.cc:2454] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7901fbacc34b54a7fffc12ca02bc84 in term 0.
I20260620 04:30:14.099406 2021 raft_consensus.cc:504] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } }
I20260620 04:30:14.099730 2021 leader_election.cc:288] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [CANDIDATE]: Term 1 election: Requested vote from peers 62779dda1c004eaa90a9a73a5b8fda0e (127.0.98.3:41053), b79d31e28880415e83c102e65623a2f4 (127.0.98.1:35725)
I20260620 04:30:14.100050 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6761637f8644e2e9147ed9fcf57dc97" candidate_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79d31e28880415e83c102e65623a2f4"
I20260620 04:30:14.100162 1665 raft_consensus.cc:3037] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:14.101188 1665 raft_consensus.cc:2454] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7901fbacc34b54a7fffc12ca02bc84 in term 1.
I20260620 04:30:14.101436 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6761637f8644e2e9147ed9fcf57dc97" candidate_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62779dda1c004eaa90a9a73a5b8fda0e"
I20260620 04:30:14.101574 1738 leader_election.cc:302] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7901fbacc34b54a7fffc12ca02bc84, b79d31e28880415e83c102e65623a2f4; no voters:
I20260620 04:30:14.101572 1945 raft_consensus.cc:3037] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:14.101745 2021 raft_consensus.cc:2781] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:14.101958 2021 raft_consensus.cc:686] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 LEADER]: Becoming Leader. State: Replica: 2b7901fbacc34b54a7fffc12ca02bc84, State: Running, Role: LEADER
I20260620 04:30:14.102281 2021 consensus_queue.cc:237] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } }
I20260620 04:30:14.102703 1945 raft_consensus.cc:2454] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7901fbacc34b54a7fffc12ca02bc84 in term 1.
I20260620 04:30:14.105217 1508 catalog_manager.cc:5495] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b7901fbacc34b54a7fffc12ca02bc84 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } health_report { overall_health: UNKNOWN } } }
I20260620 04:30:14.123099 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.119535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.0.98.2:46673)
0620 04:30:14.119649 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.119652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.119665 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:14.119891 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.119893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.119952 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.120044 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:14.120264 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.120779 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.121660 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.121893 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.121978 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.122258 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:14.122705 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:14.122795 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.122907 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.122958 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:30:14.129293 1665 raft_consensus.cc:1261] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4 [term 1 FOLLOWER]: Refusing update from remote peer 2b7901fbacc34b54a7fffc12ca02bc84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:14.129563 1945 raft_consensus.cc:1261] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e [term 1 FOLLOWER]: Refusing update from remote peer 2b7901fbacc34b54a7fffc12ca02bc84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:14.129998 2021 consensus_queue.cc:1035] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:14.130216 2024 consensus_queue.cc:1035] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:14.136869 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298784518665252864 to 7298784518565576704. Current Snapshot: MvccSnapshot[applied={T|T < 7298784518665252864}]
I20260620 04:30:14.140363 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298784518665252864 to 7298784518565576704. Current Snapshot: MvccSnapshot[applied={T|T < 7298784518665252864 or (T in {7298784518665252864})}]
I20260620 04:30:14.148522 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:14.148727 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:14.151319 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f: Sending DeleteTablet for 3 replicas of tablet a6761637f8644e2e9147ed9fcf57dc97
I20260620 04:30:14.152149 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet a6761637f8644e2e9147ed9fcf57dc97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37072
I20260620 04:30:14.152257 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet a6761637f8644e2e9147ed9fcf57dc97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35458
I20260620 04:30:14.152438 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet a6761637f8644e2e9147ed9fcf57dc97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36352
I20260620 04:30:14.152879 2039 tablet_replica.cc:330] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: stopping tablet replica
I20260620 04:30:14.153066 2038 tablet_replica.cc:330] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: stopping tablet replica
I20260620 04:30:14.153399 2039 raft_consensus.cc:2229] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:14.153313 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:30:14.153399 2038 raft_consensus.cc:2229] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:14.153748 2038 raft_consensus.cc:2258] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 04:30:14.153752 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:14.153997 2039 raft_consensus.cc:2258] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:14.154112 2037 tablet_replica.cc:330] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: stopping tablet replica
I20260620 04:30:14.154472 2037 raft_consensus.cc:2229] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:14.155861 2038 ts_tablet_manager.cc:1905] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:14.156289 2039 ts_tablet_manager.cc:1905] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:14.156330 2037 raft_consensus.cc:2258] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:14.159164 2038 ts_tablet_manager.cc:1918] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:14.159327 2038 log.cc:1196] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/a6761637f8644e2e9147ed9fcf57dc97
I20260620 04:30:14.159698 2038 ts_tablet_manager.cc:1939] T a6761637f8644e2e9147ed9fcf57dc97 P 62779dda1c004eaa90a9a73a5b8fda0e: Deleting consensus metadata
I20260620 04:30:14.160063 1785 tablet_service.cc:1467] Processing CreateTablet for tablet cd596cd9b74f4dad869e0f16b18dbfa5 (DEFAULT_TABLE table=test-table [id=2374198ce94b4b3fbc00ad90dea0fe5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:14.160344 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd596cd9b74f4dad869e0f16b18dbfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:14.160422 2039 ts_tablet_manager.cc:1918] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:14.160501 2039 log.cc:1196] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/a6761637f8644e2e9147ed9fcf57dc97
I20260620 04:30:14.160771 2039 ts_tablet_manager.cc:1939] T a6761637f8644e2e9147ed9fcf57dc97 P b79d31e28880415e83c102e65623a2f4: Deleting consensus metadata
I20260620 04:30:14.161470 1925 tablet_service.cc:1467] Processing CreateTablet for tablet cd596cd9b74f4dad869e0f16b18dbfa5 (DEFAULT_TABLE table=test-table [id=2374198ce94b4b3fbc00ad90dea0fe5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:14.161470 1645 tablet_service.cc:1467] Processing CreateTablet for tablet cd596cd9b74f4dad869e0f16b18dbfa5 (DEFAULT_TABLE table=test-table [id=2374198ce94b4b3fbc00ad90dea0fe5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:14.161703 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd596cd9b74f4dad869e0f16b18dbfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:14.161703 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd596cd9b74f4dad869e0f16b18dbfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:14.162755 2016 tablet_bootstrap.cc:492] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: Bootstrap starting.
I20260620 04:30:14.162967 1494 catalog_manager.cc:4841] TS b79d31e28880415e83c102e65623a2f4 (127.0.98.1:35725): tablet a6761637f8644e2e9147ed9fcf57dc97 (table test-table [id=46fb7974c11944528011e32f8b3bded0]) successfully deleted
I20260620 04:30:14.163268 1493 catalog_manager.cc:4841] TS 62779dda1c004eaa90a9a73a5b8fda0e (127.0.98.3:41053): tablet a6761637f8644e2e9147ed9fcf57dc97 (table test-table [id=46fb7974c11944528011e32f8b3bded0]) successfully deleted
I20260620 04:30:14.163686 2016 tablet_bootstrap.cc:654] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:14.163777 2014 tablet_bootstrap.cc:492] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: Bootstrap starting.
I20260620 04:30:14.164691 2014 tablet_bootstrap.cc:654] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:14.165297 2015 tablet_bootstrap.cc:492] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: Bootstrap starting.
I20260620 04:30:14.165356 2037 ts_tablet_manager.cc:1905] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:14.166463 2015 tablet_bootstrap.cc:654] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:14.167723 2014 tablet_bootstrap.cc:492] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: No bootstrap required, opened a new log
I20260620 04:30:14.167815 2014 ts_tablet_manager.cc:1397] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 04:30:14.167850 2037 ts_tablet_manager.cc:1918] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:14.167933 2037 log.cc:1196] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/a6761637f8644e2e9147ed9fcf57dc97
I20260620 04:30:14.168202 2037 ts_tablet_manager.cc:1939] T a6761637f8644e2e9147ed9fcf57dc97 P 2b7901fbacc34b54a7fffc12ca02bc84: Deleting consensus metadata
I20260620 04:30:14.168272 2014 raft_consensus.cc:348] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.168381 2014 raft_consensus.cc:374] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:14.168416 2014 raft_consensus.cc:729] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62779dda1c004eaa90a9a73a5b8fda0e, State: Initialized, Role: FOLLOWER
I20260620 04:30:14.168546 2014 consensus_queue.cc:260] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.168785 2014 ts_tablet_manager.cc:1428] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:14.169564 1493 catalog_manager.cc:4841] TS 2b7901fbacc34b54a7fffc12ca02bc84 (127.0.98.2:46673): tablet a6761637f8644e2e9147ed9fcf57dc97 (table test-table [id=46fb7974c11944528011e32f8b3bded0]) successfully deleted
I20260620 04:30:14.169919 2016 tablet_bootstrap.cc:492] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: No bootstrap required, opened a new log
I20260620 04:30:14.170001 2016 ts_tablet_manager.cc:1397] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260620 04:30:14.170176 2015 tablet_bootstrap.cc:492] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: No bootstrap required, opened a new log
I20260620 04:30:14.170259 2015 ts_tablet_manager.cc:1397] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 04:30:14.170410 2016 raft_consensus.cc:348] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.170509 2016 raft_consensus.cc:374] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:14.170535 2016 raft_consensus.cc:729] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7901fbacc34b54a7fffc12ca02bc84, State: Initialized, Role: FOLLOWER
I20260620 04:30:14.170657 2016 consensus_queue.cc:260] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.170884 2016 ts_tablet_manager.cc:1428] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:30:14.170961 2015 raft_consensus.cc:348] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.171069 2015 raft_consensus.cc:374] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:14.171103 2015 raft_consensus.cc:729] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79d31e28880415e83c102e65623a2f4, State: Initialized, Role: FOLLOWER
I20260620 04:30:14.171262 2015 consensus_queue.cc:260] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.171530 2015 ts_tablet_manager.cc:1428] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260620 04:30:14.242035 1992 tablet.cc:2369] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:30:14.276574 1852 tablet.cc:2369] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:30:14.318234 1712 tablet.cc:2369] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:30:14.480432 2022 raft_consensus.cc:482] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:14.480589 2022 raft_consensus.cc:504] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.481732 2022 leader_election.cc:288] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b7901fbacc34b54a7fffc12ca02bc84 (127.0.98.2:46673), 62779dda1c004eaa90a9a73a5b8fda0e (127.0.98.3:41053)
I20260620 04:30:14.486702 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.482517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55315 (local address 127.0.98.3:41053)
0620 04:30:14.482629 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.482633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.482892 (+ 259us) server_negotiation.cc:407] Connection header received
0620 04:30:14.482959 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.482962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.483012 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.483109 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:14.483329 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.484001 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.485480 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.486079 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.486191 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.486255 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.486512 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.486574 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:30:14.486702 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.482160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52737 (local address 127.0.98.2:46673)
0620 04:30:14.482289 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.482293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.482361 (+ 68us) server_negotiation.cc:407] Connection header received
0620 04:30:14.482549 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.482553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.482610 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.482676 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:14.483009 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.483833 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.485479 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.486085 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.486188 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.486308 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.486489 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.486566 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:30:14.486763 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.482383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41053 (local address 127.0.98.1:55315)
0620 04:30:14.482767 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:14.482779 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:14.482871 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:14.483121 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:14.483123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:14.483138 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:14.483235 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.483240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.484100 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.484103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.485357 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.485369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.486255 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.486334 (+ 79us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.486410 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.486427 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.486488 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.486560 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":20,"thread_start_us":95,"threads_started":1}
I20260620 04:30:14.486763 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.482004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46673 (local address 127.0.98.1:52737)
0620 04:30:14.482259 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:14.482276 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:14.482414 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:14.482722 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:14.482725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:14.482746 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:14.482883 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.482889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.483956 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.483959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.485357 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.485369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.486252 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.486257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.486362 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.486389 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.486468 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.486571 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":66,"thread_start_us":86,"threads_started":1}
I20260620 04:30:14.487290 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd596cd9b74f4dad869e0f16b18dbfa5" candidate_uuid: "b79d31e28880415e83c102e65623a2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" is_pre_election: true
I20260620 04:30:14.487350 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd596cd9b74f4dad869e0f16b18dbfa5" candidate_uuid: "b79d31e28880415e83c102e65623a2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" is_pre_election: true
I20260620 04:30:14.487448 1945 raft_consensus.cc:2454] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79d31e28880415e83c102e65623a2f4 in term 0.
I20260620 04:30:14.487589 1805 raft_consensus.cc:2454] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79d31e28880415e83c102e65623a2f4 in term 0.
I20260620 04:30:14.487852 1597 leader_election.cc:302] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62779dda1c004eaa90a9a73a5b8fda0e, b79d31e28880415e83c102e65623a2f4; no voters:
I20260620 04:30:14.488137 2022 raft_consensus.cc:2781] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:14.488206 2022 raft_consensus.cc:482] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:14.488236 2022 raft_consensus.cc:3037] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:14.489311 2022 raft_consensus.cc:504] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.489666 2022 leader_election.cc:288] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2b7901fbacc34b54a7fffc12ca02bc84 (127.0.98.2:46673), 62779dda1c004eaa90a9a73a5b8fda0e (127.0.98.3:41053)
I20260620 04:30:14.490067 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd596cd9b74f4dad869e0f16b18dbfa5" candidate_uuid: "b79d31e28880415e83c102e65623a2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7901fbacc34b54a7fffc12ca02bc84"
I20260620 04:30:14.490118 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd596cd9b74f4dad869e0f16b18dbfa5" candidate_uuid: "b79d31e28880415e83c102e65623a2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62779dda1c004eaa90a9a73a5b8fda0e"
I20260620 04:30:14.490181 1805 raft_consensus.cc:3037] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:14.490247 1945 raft_consensus.cc:3037] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:14.491205 1805 raft_consensus.cc:2454] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79d31e28880415e83c102e65623a2f4 in term 1.
I20260620 04:30:14.491214 1945 raft_consensus.cc:2454] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79d31e28880415e83c102e65623a2f4 in term 1.
I20260620 04:30:14.491508 1597 leader_election.cc:302] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7901fbacc34b54a7fffc12ca02bc84, b79d31e28880415e83c102e65623a2f4; no voters:
I20260620 04:30:14.491721 2022 raft_consensus.cc:2781] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:14.491977 2022 raft_consensus.cc:686] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 1 LEADER]: Becoming Leader. State: Replica: b79d31e28880415e83c102e65623a2f4, State: Running, Role: LEADER
I20260620 04:30:14.492268 2022 consensus_queue.cc:237] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.494990 1508 catalog_manager.cc:5495] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b79d31e28880415e83c102e65623a2f4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b79d31e28880415e83c102e65623a2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } health_report { overall_health: UNKNOWN } } }
Jun 20 04:30:14 dist-test-slave-rzqm krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:14.551362 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.543067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.0.98.62:38529)
0620 04:30:14.543357 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.543365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.543388 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:30:14.543542 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.543545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.543598 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.543701 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:14.544360 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.544912 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.545706 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.545947 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.546032 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.548297 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:14.548301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:14.548304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:14.548327 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:14.549715 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:14.550157 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:14.550161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:14.550163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:14.550242 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:14.550467 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:14.550470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:14.550472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:14.550650 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:14.550762 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.551049 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.551192 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":218,"thread_start_us":111,"threads_started":1}
I20260620 04:30:14.551362 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.543004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38529 (local address 127.0.0.1:51336)
0620 04:30:14.543320 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:14.543336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:14.543442 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:14.543723 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:14.543725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:14.544139 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:14.544244 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.544251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.545027 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.545031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.545604 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.545611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.546045 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.546048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.546152 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.546706 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:14.546718 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:14.548162 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:14.549812 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:14.549823 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:14.549829 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:14.550073 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:14.550316 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:14.550319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:14.550321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:14.550395 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:14.550744 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:14.550772 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:14.550888 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.550984 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.551145 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":19,"thread_start_us":138,"threads_started":1}
I20260620 04:30:14.555348 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51336:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 04:30:14.556082 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:14.560864 1785 tablet_service.cc:1467] Processing CreateTablet for tablet bc7c6b18ac34426e86320087405fe5ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=750846c40e174862960afbaf88a48866]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:30:14.560842 1645 tablet_service.cc:1467] Processing CreateTablet for tablet bc7c6b18ac34426e86320087405fe5ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=750846c40e174862960afbaf88a48866]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:30:14.560963 1925 tablet_service.cc:1467] Processing CreateTablet for tablet bc7c6b18ac34426e86320087405fe5ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=750846c40e174862960afbaf88a48866]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:30:14.561259 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7c6b18ac34426e86320087405fe5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:14.561275 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7c6b18ac34426e86320087405fe5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:14.561268 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7c6b18ac34426e86320087405fe5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:14.563488 2016 tablet_bootstrap.cc:492] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: Bootstrap starting.
I20260620 04:30:14.563998 2014 tablet_bootstrap.cc:492] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: Bootstrap starting.
I20260620 04:30:14.564393 2016 tablet_bootstrap.cc:654] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:14.564965 2014 tablet_bootstrap.cc:654] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:14.565496 2016 tablet_bootstrap.cc:492] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: No bootstrap required, opened a new log
I20260620 04:30:14.565570 2015 tablet_bootstrap.cc:492] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: Bootstrap starting.
I20260620 04:30:14.565586 2016 ts_tablet_manager.cc:1397] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:14.565991 2016 raft_consensus.cc:348] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.566110 2016 raft_consensus.cc:374] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:14.566140 2016 raft_consensus.cc:729] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7901fbacc34b54a7fffc12ca02bc84, State: Initialized, Role: FOLLOWER
I20260620 04:30:14.566270 2016 consensus_queue.cc:260] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.566414 2015 tablet_bootstrap.cc:654] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:14.566509 2033 tablet_replica.cc:439] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } }
I20260620 04:30:14.566681 2033 tablet_replica.cc:442] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:14.567565 2015 tablet_bootstrap.cc:492] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: No bootstrap required, opened a new log
I20260620 04:30:14.567631 2015 ts_tablet_manager.cc:1397] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:14.567646 2014 tablet_bootstrap.cc:492] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: No bootstrap required, opened a new log
I20260620 04:30:14.567725 2014 ts_tablet_manager.cc:1397] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 04:30:14.567983 2015 raft_consensus.cc:348] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.568074 2015 raft_consensus.cc:374] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:14.568101 2015 raft_consensus.cc:729] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79d31e28880415e83c102e65623a2f4, State: Initialized, Role: FOLLOWER
I20260620 04:30:14.568230 2015 consensus_queue.cc:260] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.568239 2014 raft_consensus.cc:348] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.568347 2014 raft_consensus.cc:374] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:14.568378 2014 raft_consensus.cc:729] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62779dda1c004eaa90a9a73a5b8fda0e, State: Initialized, Role: FOLLOWER
I20260620 04:30:14.568445 2015 ts_tablet_manager.cc:1428] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:14.568488 2014 consensus_queue.cc:260] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.568626 2022 tablet_replica.cc:439] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } }
I20260620 04:30:14.568722 2022 tablet_replica.cc:442] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:14.568770 2020 tablet_replica.cc:439] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } }
I20260620 04:30:14.568914 2020 tablet_replica.cc:442] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:14.568775 2014 ts_tablet_manager.cc:1428] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 04:30:14.566520 2016 ts_tablet_manager.cc:1428] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:14.603940 2022 raft_consensus.cc:482] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:14.604116 2022 raft_consensus.cc:504] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.604517 2022 leader_election.cc:288] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b7901fbacc34b54a7fffc12ca02bc84 (127.0.98.2:46673), 62779dda1c004eaa90a9a73a5b8fda0e (127.0.98.3:41053)
I20260620 04:30:14.605029 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7c6b18ac34426e86320087405fe5ff" candidate_uuid: "b79d31e28880415e83c102e65623a2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" is_pre_election: true
I20260620 04:30:14.605082 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7c6b18ac34426e86320087405fe5ff" candidate_uuid: "b79d31e28880415e83c102e65623a2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" is_pre_election: true
I20260620 04:30:14.605196 1805 raft_consensus.cc:2454] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79d31e28880415e83c102e65623a2f4 in term 0.
I20260620 04:30:14.605270 1945 raft_consensus.cc:2454] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b79d31e28880415e83c102e65623a2f4 in term 0.
I20260620 04:30:14.605561 1597 leader_election.cc:302] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7901fbacc34b54a7fffc12ca02bc84, b79d31e28880415e83c102e65623a2f4; no voters:
I20260620 04:30:14.605744 2022 raft_consensus.cc:2781] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:14.605805 2022 raft_consensus.cc:482] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:14.605844 2022 raft_consensus.cc:3037] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:14.606866 2022 raft_consensus.cc:504] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.607182 2022 leader_election.cc:288] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2b7901fbacc34b54a7fffc12ca02bc84 (127.0.98.2:46673), 62779dda1c004eaa90a9a73a5b8fda0e (127.0.98.3:41053)
I20260620 04:30:14.607527 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7c6b18ac34426e86320087405fe5ff" candidate_uuid: "b79d31e28880415e83c102e65623a2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7901fbacc34b54a7fffc12ca02bc84"
I20260620 04:30:14.607554 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7c6b18ac34426e86320087405fe5ff" candidate_uuid: "b79d31e28880415e83c102e65623a2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62779dda1c004eaa90a9a73a5b8fda0e"
I20260620 04:30:14.607654 1805 raft_consensus.cc:3037] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:14.607671 1945 raft_consensus.cc:3037] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:14.608731 1945 raft_consensus.cc:2454] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79d31e28880415e83c102e65623a2f4 in term 1.
I20260620 04:30:14.608770 1805 raft_consensus.cc:2454] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b79d31e28880415e83c102e65623a2f4 in term 1.
I20260620 04:30:14.609036 1597 leader_election.cc:302] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7901fbacc34b54a7fffc12ca02bc84, b79d31e28880415e83c102e65623a2f4; no voters:
I20260620 04:30:14.609196 2022 raft_consensus.cc:2781] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:14.609304 2022 raft_consensus.cc:686] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [term 1 LEADER]: Becoming Leader. State: Replica: b79d31e28880415e83c102e65623a2f4, State: Running, Role: LEADER
I20260620 04:30:14.609431 2022 consensus_queue.cc:237] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } }
I20260620 04:30:14.610081 2045 tablet_replica.cc:439] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: TxnStatusTablet state changed. Reason: New leader b79d31e28880415e83c102e65623a2f4. Latest consensus state: current_term: 1 leader_uuid: "b79d31e28880415e83c102e65623a2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } }
I20260620 04:30:14.610256 2045 tablet_replica.cc:442] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:30:14.610567 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 04:30:14.610733 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 04:30:14.611070 1506 catalog_manager.cc:5495] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b79d31e28880415e83c102e65623a2f4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b79d31e28880415e83c102e65623a2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } health_report { overall_health: UNKNOWN } } }
I20260620 04:30:14.627084 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.622470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.0.98.1:35725)
0620 04:30:14.622728 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.622734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.622749 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:14.622911 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.622914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.622969 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.623070 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:14.623319 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.624178 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.625434 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.625681 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.625769 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.626093 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:14.626621 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:14.626719 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.626844 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.626898 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
I20260620 04:30:14.627074 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.622396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35725 (local address 127.0.0.1:36362)
0620 04:30:14.622636 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:14.622651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:14.622763 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:14.623093 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:14.623095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:14.623103 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:14.623215 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.623220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.624278 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.624282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.625305 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.625316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.625780 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.625783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.625881 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.625934 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:14.626726 (+ 792us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:14.626742 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.626825 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.626930 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 04:30:14.629940 1945 raft_consensus.cc:1261] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e [term 1 FOLLOWER]: Refusing update from remote peer b79d31e28880415e83c102e65623a2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:14.630090 1805 raft_consensus.cc:1261] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 FOLLOWER]: Refusing update from remote peer b79d31e28880415e83c102e65623a2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:14.630405 2045 consensus_queue.cc:1035] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:14.630575 2022 consensus_queue.cc:1035] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:14.632012 2020 tablet_replica.cc:439] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: TxnStatusTablet state changed. Reason: New leader b79d31e28880415e83c102e65623a2f4. Latest consensus state: current_term: 1 leader_uuid: "b79d31e28880415e83c102e65623a2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } }
I20260620 04:30:14.632131 2020 tablet_replica.cc:442] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:14.633394 2033 tablet_replica.cc:439] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: TxnStatusTablet state changed. Reason: New leader b79d31e28880415e83c102e65623a2f4. Latest consensus state: current_term: 1 leader_uuid: "b79d31e28880415e83c102e65623a2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } }
I20260620 04:30:14.633507 2033 tablet_replica.cc:442] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:14.634941 2045 tablet_replica.cc:439] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b79d31e28880415e83c102e65623a2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } }
I20260620 04:30:14.635051 2045 tablet_replica.cc:442] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:30:14.635138 2022 tablet_replica.cc:439] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b79d31e28880415e83c102e65623a2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } }
I20260620 04:30:14.635243 2022 tablet_replica.cc:442] T bc7c6b18ac34426e86320087405fe5ff P b79d31e28880415e83c102e65623a2f4: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:30:14.635494 2020 tablet_replica.cc:439] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b79d31e28880415e83c102e65623a2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } }
I20260620 04:30:14.635591 2020 tablet_replica.cc:442] T bc7c6b18ac34426e86320087405fe5ff P 62779dda1c004eaa90a9a73a5b8fda0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:14.636080 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298784520719040512 to 7298784520641626112. Current Snapshot: MvccSnapshot[applied={T|T < 7298784520719040512 or (T in {7298784520719040512})}]
I20260620 04:30:14.637347 2033 tablet_replica.cc:439] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b79d31e28880415e83c102e65623a2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b79d31e28880415e83c102e65623a2f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35725 } } peers { permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 } } peers { permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 } } }
I20260620 04:30:14.637466 2033 tablet_replica.cc:442] T bc7c6b18ac34426e86320087405fe5ff P 2b7901fbacc34b54a7fffc12ca02bc84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:14.644372 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.640579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.0.98.1:35725)
0620 04:30:14.640683 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.640687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.640700 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:14.640870 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.640890 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.640937 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.641056 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:14.641425 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.642161 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.643218 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.643504 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.643596 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.643853 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:14.643923 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:14.643977 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.644066 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.644117 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:30:14.656687 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.652821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37807 (local address 127.0.98.1:35725)
0620 04:30:14.652931 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.652935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.653263 (+ 328us) server_negotiation.cc:407] Connection header received
0620 04:30:14.653421 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.653424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.653459 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.653509 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:14.653737 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.654218 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.655480 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.655803 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.655908 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.656190 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:14.656274 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:14.656337 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.656473 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.656525 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:30:14.657235 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.652715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35725 (local address 127.0.98.1:37807)
0620 04:30:14.653191 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:14.653206 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:14.653316 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:14.653531 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:14.653534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:14.653541 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:14.653650 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.653655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.654309 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.654312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.655348 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:14.655360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.655906 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.655910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:14.656005 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.656060 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:14.656360 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:14.656368 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 04:30:14.656458 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 04:30:14.656544 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260620 04:30:14.665899 1805 raft_consensus.cc:1261] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 FOLLOWER]: Refusing update from remote peer b79d31e28880415e83c102e65623a2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:14.665915 1945 raft_consensus.cc:1261] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e [term 1 FOLLOWER]: Refusing update from remote peer b79d31e28880415e83c102e65623a2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:14.666364 2022 consensus_queue.cc:1035] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7901fbacc34b54a7fffc12ca02bc84" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:14.666648 2064 consensus_queue.cc:1035] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62779dda1c004eaa90a9a73a5b8fda0e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:14.705675 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:14.705854 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:14.708004 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b213cf8533e4b2485b0070b381fca8f: Sending DeleteTablet for 3 replicas of tablet cd596cd9b74f4dad869e0f16b18dbfa5
I20260620 04:30:14.708604 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet cd596cd9b74f4dad869e0f16b18dbfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36352
I20260620 04:30:14.708885 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet cd596cd9b74f4dad869e0f16b18dbfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37072
I20260620 04:30:14.709030 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet cd596cd9b74f4dad869e0f16b18dbfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35458
I20260620 04:30:14.709256 2074 tablet_replica.cc:330] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: stopping tablet replica
I20260620 04:30:14.709323 2075 tablet_replica.cc:330] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: stopping tablet replica
I20260620 04:30:14.709447 2075 raft_consensus.cc:2229] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:14.709479 2074 raft_consensus.cc:2229] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:14.709568 2075 raft_consensus.cc:2258] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:14.709941 2074 raft_consensus.cc:2258] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:14.710191 2075 ts_tablet_manager.cc:1905] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:14.711059 2074 ts_tablet_manager.cc:1905] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:14.713285 2075 ts_tablet_manager.cc:1918] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:14.713366 2075 log.cc:1196] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/cd596cd9b74f4dad869e0f16b18dbfa5
I20260620 04:30:14.713480 2074 ts_tablet_manager.cc:1918] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:14.713559 2074 log.cc:1196] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/cd596cd9b74f4dad869e0f16b18dbfa5
I20260620 04:30:14.713614 2075 ts_tablet_manager.cc:1939] T cd596cd9b74f4dad869e0f16b18dbfa5 P 62779dda1c004eaa90a9a73a5b8fda0e: Deleting consensus metadata
I20260620 04:30:14.713825 2074 ts_tablet_manager.cc:1939] T cd596cd9b74f4dad869e0f16b18dbfa5 P 2b7901fbacc34b54a7fffc12ca02bc84: Deleting consensus metadata
I20260620 04:30:14.714376 1493 catalog_manager.cc:4841] TS 62779dda1c004eaa90a9a73a5b8fda0e (127.0.98.3:41053): tablet cd596cd9b74f4dad869e0f16b18dbfa5 (table test-table [id=2374198ce94b4b3fbc00ad90dea0fe5b]) successfully deleted
I20260620 04:30:14.714594 1493 catalog_manager.cc:4841] TS 2b7901fbacc34b54a7fffc12ca02bc84 (127.0.98.2:46673): tablet cd596cd9b74f4dad869e0f16b18dbfa5 (table test-table [id=2374198ce94b4b3fbc00ad90dea0fe5b]) successfully deleted
I20260620 04:30:14.714603 2073 tablet_replica.cc:330] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: stopping tablet replica
I20260620 04:30:14.714727 2073 raft_consensus.cc:2229] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:14.715008 2073 raft_consensus.cc:2258] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:14.715565 2073 ts_tablet_manager.cc:1905] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:14.718308 2073 ts_tablet_manager.cc:1918] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:14.718400 2073 log.cc:1196] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/cd596cd9b74f4dad869e0f16b18dbfa5
I20260620 04:30:14.718658 2073 ts_tablet_manager.cc:1939] T cd596cd9b74f4dad869e0f16b18dbfa5 P b79d31e28880415e83c102e65623a2f4: Deleting consensus metadata
I20260620 04:30:14.719394 1494 catalog_manager.cc:4841] TS b79d31e28880415e83c102e65623a2f4 (127.0.98.1:35725): tablet cd596cd9b74f4dad869e0f16b18dbfa5 (table test-table [id=2374198ce94b4b3fbc00ad90dea0fe5b]) successfully deleted
Jun 20 04:30:14 dist-test-slave-rzqm krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929814, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:30:14.725621 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.710775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.0.98.1:35725)
0620 04:30:14.710888 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.710892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.714447 (+ 3555us) server_negotiation.cc:407] Connection header received
0620 04:30:14.714632 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.714635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.714693 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.714741 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:14.715645 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.716345 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.717564 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.717947 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.718071 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.721865 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:14.721875 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:14.721891 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:14.721924 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:14.723644 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:14.724095 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:14.724101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:14.724102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:14.724185 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:14.724457 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:14.724461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:14.724462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:14.724723 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:14.724946 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.725375 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.725494 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 04:30:14.726507 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36376
I20260620 04:30:14.735855 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:14.728461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.0.98.62:38529)
0620 04:30:14.728580 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:14.728584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:14.728684 (+ 100us) server_negotiation.cc:407] Connection header received
0620 04:30:14.728799 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:14.728802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:14.728853 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:14.728915 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:14.729623 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.730173 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.731041 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:14.731330 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:14.731421 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:14.732893 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:14.732898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:14.732900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:14.732929 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:14.734303 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:14.734700 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:14.734704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:14.734706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:14.734787 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:14.735025 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:14.735029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:14.735031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:14.735219 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:14.735339 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:14.735561 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 04:30:14.735694 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 04:30:14.736268 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51344
I20260620 04:30:14.737605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 1580
I20260620 04:30:14.744073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 1720
I20260620 04:30:14.749871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 1860
I20260620 04:30:14.754832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 1476
2026-06-20T04:30:14Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2757 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:14 dist-test-slave-rzqm krb5kdc[2082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:14 dist-test-slave-rzqm krb5kdc[2082](info): set up 2 sockets
Jun 20 04:30:14 dist-test-slave-rzqm krb5kdc[2082](info): commencing operation
krb5kdc: starting...
W20260620 04:30:15.704694 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:15 dist-test-slave-rzqm krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:15.819546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44581
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:15.928949 2095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:15.929255 2095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:15.929304 2095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:15.933506 2095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:15.933584 2095 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 04:30:15.933600 2095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:15.933619 2095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:15.933633 2095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:15.933648 2095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:15.939113 2095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44581
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:15.940191 2095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:15.941205 2095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:15.949690 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:15.949683 2103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:15.949869 2095 server_base.cc:1034] running on GCE node
W20260620 04:30:15.950021 2101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:15.950654 2095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:15.951843 2095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:15.953048 2095 hybrid_clock.cc:648] HybridClock initialized: now 1781929815953017 us; error 47 us; skew 500 ppm
Jun 20 04:30:15 dist-test-slave-rzqm krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:15.955907 2095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:15.956869 2095 webserver.cc:460] Webserver started at http://127.0.98.62:43589/ using document root <none> and password file <none>
I20260620 04:30:15.957470 2095 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:15.957544 2095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:15.957773 2095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:15.959889 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "1f5a905b3dcd48559d1a3ea6d3877b78"
format_stamp: "Formatted at 2026-06-20 04:30:15 on dist-test-slave-rzqm"
server_key: "c8ba3f89b8520eb669c99844c98adeba"
server_key_iv: "7a5e976e915cffd81d26634b661ea403"
server_key_version: "encryptionkey@0"
I20260620 04:30:15.960443 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f5a905b3dcd48559d1a3ea6d3877b78"
format_stamp: "Formatted at 2026-06-20 04:30:15 on dist-test-slave-rzqm"
server_key: "c8ba3f89b8520eb669c99844c98adeba"
server_key_iv: "7a5e976e915cffd81d26634b661ea403"
server_key_version: "encryptionkey@0"
I20260620 04:30:15.964228 2095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:30:15.966718 2110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:15.967988 2095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 04:30:15.968107 2095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "1f5a905b3dcd48559d1a3ea6d3877b78"
format_stamp: "Formatted at 2026-06-20 04:30:15 on dist-test-slave-rzqm"
server_key: "c8ba3f89b8520eb669c99844c98adeba"
server_key_iv: "7a5e976e915cffd81d26634b661ea403"
server_key_version: "encryptionkey@0"
I20260620 04:30:15.968210 2095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:15.982883 2095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:15.986166 2095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:15.986351 2095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:15.994068 2095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44581
I20260620 04:30:15.994138 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44581 every 8 connection(s)
I20260620 04:30:15.995107 2095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:15.995836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 2095
I20260620 04:30:15.995956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:15.996289 392 external_mini_cluster.cc:1428] Setting key e29015a39278249c43e3b26ee3a0f490
I20260620 04:30:15.998481 2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:16.004982 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78: Bootstrap starting.
Jun 20 04:30:16 dist-test-slave-rzqm krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:16.007309 2173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:16.008428 2173 log.cc:824] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:16.010677 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:15.997749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.98.62:44581)
0620 04:30:15.998179 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:15.998194 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:15.998231 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:30:15.998946 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:15.998967 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:15.999308 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:15.999608 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:16.000483 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.001578 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.002731 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.003194 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.003327 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.005973 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:16.005978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:16.005991 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:16.006020 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:16.007716 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.008146 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.008151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.008156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.008274 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.008535 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.008539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.008540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.009078 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:16.009281 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.009575 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.009882 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":137,"threads_started":1}
I20260620 04:30:16.010732 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78: No bootstrap required, opened a new log
I20260620 04:30:16.014228 2173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5a905b3dcd48559d1a3ea6d3877b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44581 } }
I20260620 04:30:16.014485 2173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:16.014554 2173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5a905b3dcd48559d1a3ea6d3877b78, State: Initialized, Role: FOLLOWER
I20260620 04:30:16.014964 2173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5a905b3dcd48559d1a3ea6d3877b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44581 } }
I20260620 04:30:16.015091 2173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:16.015156 2173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:16.015257 2173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:16.016201 2173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5a905b3dcd48559d1a3ea6d3877b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44581 } }
I20260620 04:30:16.016544 2173 leader_election.cc:302] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f5a905b3dcd48559d1a3ea6d3877b78; no voters:
I20260620 04:30:16.016889 2173 leader_election.cc:288] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:16.016989 2178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:16.017230 2178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [term 1 LEADER]: Becoming Leader. State: Replica: 1f5a905b3dcd48559d1a3ea6d3877b78, State: Running, Role: LEADER
I20260620 04:30:16.017560 2178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5a905b3dcd48559d1a3ea6d3877b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44581 } }
I20260620 04:30:16.018123 2173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:16.019186 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f5a905b3dcd48559d1a3ea6d3877b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5a905b3dcd48559d1a3ea6d3877b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44581 } } }
I20260620 04:30:16.019218 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f5a905b3dcd48559d1a3ea6d3877b78. Latest consensus state: current_term: 1 leader_uuid: "1f5a905b3dcd48559d1a3ea6d3877b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5a905b3dcd48559d1a3ea6d3877b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44581 } } }
I20260620 04:30:16.019315 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:16.019315 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:16.019694 2187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:16.022512 2187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:16.028010 2187 catalog_manager.cc:1347] Generated new cluster ID: 70030f59f87e449991016556804b47e3
I20260620 04:30:16.028077 2187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:16.042836 2187 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:16.043823 2187 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:16.050634 2187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78: Generated new TSK 0
I20260620 04:30:16.051497 2187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:30:16.078115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44581
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:16.204808 2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:16.205130 2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:16.205196 2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:16.209317 2199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:16.209405 2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:16.209450 2199 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 04:30:16.209468 2199 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:30:16.209514 2199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:16.215087 2199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:16.216197 2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:16.217278 2199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:16.227227 2205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:16.227326 2199 server_base.cc:1034] running on GCE node
W20260620 04:30:16.227250 2207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:16.227240 2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:16.227977 2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:16.228614 2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:16.229777 2199 hybrid_clock.cc:648] HybridClock initialized: now 1781929816229749 us; error 43 us; skew 500 ppm
Jun 20 04:30:16 dist-test-slave-rzqm krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:16.232658 2199 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:30:16.233649 2199 webserver.cc:460] Webserver started at http://127.0.98.1:34775/ using document root <none> and password file <none>
I20260620 04:30:16.234192 2199 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:16.234259 2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:16.234478 2199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:16.236545 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "062b3f3e9c124b8a9e25c830f95de006"
format_stamp: "Formatted at 2026-06-20 04:30:16 on dist-test-slave-rzqm"
server_key: "e14fbfd5fdb80da5d05776f065734caf"
server_key_iv: "a233687d42b318a27e9fd1a48e7db934"
server_key_version: "encryptionkey@0"
I20260620 04:30:16.237100 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "062b3f3e9c124b8a9e25c830f95de006"
format_stamp: "Formatted at 2026-06-20 04:30:16 on dist-test-slave-rzqm"
server_key: "e14fbfd5fdb80da5d05776f065734caf"
server_key_iv: "a233687d42b318a27e9fd1a48e7db934"
server_key_version: "encryptionkey@0"
I20260620 04:30:16.240877 2199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 04:30:16.243335 2214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:16.244596 2199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:16.244714 2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "062b3f3e9c124b8a9e25c830f95de006"
format_stamp: "Formatted at 2026-06-20 04:30:16 on dist-test-slave-rzqm"
server_key: "e14fbfd5fdb80da5d05776f065734caf"
server_key_iv: "a233687d42b318a27e9fd1a48e7db934"
server_key_version: "encryptionkey@0"
I20260620 04:30:16.244812 2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:16.283277 2199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:16.286643 2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:16.286846 2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:16.288390 2199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:16.288451 2199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:16.288508 2199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:16.288538 2199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:30:16 dist-test-slave-rzqm krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:16.298727 2199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40331
I20260620 04:30:16.298878 2329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40331 every 8 connection(s)
I20260620 04:30:16.299674 2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:16.302232 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.289355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36241 (local address 127.0.98.62:44581)
0620 04:30:16.289513 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.289517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.290691 (+ 1174us) server_negotiation.cc:407] Connection header received
0620 04:30:16.291744 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.291758 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.291829 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.291939 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:16.293434 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.294279 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.295670 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.295911 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.296007 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.298634 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:16.298638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:16.298641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:16.298673 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:16.300122 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.300679 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.300682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.300684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.300770 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.301099 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.301103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.301105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.301300 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:16.301387 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.301931 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.302068 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 04:30:16.303001 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.289989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44581 (local address 127.0.98.1:36241)
0620 04:30:16.290525 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.290572 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.291497 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.292171 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.292181 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.292574 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:16.293275 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.293292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.294425 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.294430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.295533 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.295545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.296060 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.296065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.296270 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.296878 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:16.296913 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:16.298443 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:16.300259 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.300265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.300280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.300581 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.300855 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.300857 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.300859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.300987 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.301428 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:16.301436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:16.301682 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.301844 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.302177 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":127,"threads_started":1}
I20260620 04:30:16.304303 2330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44581
I20260620 04:30:16.304450 2330 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:16.304970 2330 heartbeater.cc:511] Master 127.0.98.62:44581 requested a full tablet report, sending...
I20260620 04:30:16.306510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 2199
I20260620 04:30:16.306620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:16.306612 2127 ts_manager.cc:194] Registered new tserver with Master: 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331)
I20260620 04:30:16.306885 392 external_mini_cluster.cc:1428] Setting key cb6595ffd792278ffa7d5cda4f596685
I20260620 04:30:16.307957 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36241
I20260620 04:30:16.314953 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.306326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44581 (local address 127.0.98.1:53635)
0620 04:30:16.306770 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.306795 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.306893 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.307135 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.307138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.307358 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:16.307480 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.307487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.308421 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.308424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.309225 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.309235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.310944 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.310947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.311023 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.311547 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:16.311558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:16.311952 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:16.313604 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.313608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.313610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.313798 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.314042 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.314045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.314047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.314124 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.314451 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:16.314454 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:16.314542 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.314624 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.314763 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":85,"threads_started":1}
I20260620 04:30:16.314946 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.306487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53635 (local address 127.0.98.62:44581)
0620 04:30:16.306609 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.306613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.306868 (+ 255us) server_negotiation.cc:407] Connection header received
0620 04:30:16.306971 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.306973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.307021 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.307103 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:16.307574 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.308299 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.310505 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.310827 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.310933 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.312082 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:16.312086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:16.312089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:16.312113 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:16.313504 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.313881 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.313885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.313886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.313963 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.314198 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.314201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.314203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.314365 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:16.314445 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.314684 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.314786 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:30:16.332106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44581
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:16.440925 2339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:16.441267 2339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:16.441336 2339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:16.445382 2339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:16.445448 2339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:16.445506 2339 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 04:30:16.445528 2339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:30:16.445577 2339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:16.450922 2339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:16.452009 2339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:16.453078 2339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:16.462958 2344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:16.463023 2339 server_base.cc:1034] running on GCE node
W20260620 04:30:16.463006 2347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:16.462944 2345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:16.463704 2339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:16.464435 2339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:16.465620 2339 hybrid_clock.cc:648] HybridClock initialized: now 1781929816465594 us; error 36 us; skew 500 ppm
Jun 20 04:30:16 dist-test-slave-rzqm krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:16.468482 2339 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:30:16.469452 2339 webserver.cc:460] Webserver started at http://127.0.98.2:37831/ using document root <none> and password file <none>
I20260620 04:30:16.470007 2339 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:16.470078 2339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:16.470373 2339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:16.472725 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "fc8b42c0376847348fdbf281a7a986f7"
format_stamp: "Formatted at 2026-06-20 04:30:16 on dist-test-slave-rzqm"
server_key: "00d27303ab1304179c3cce255a3aa457"
server_key_iv: "cb3b1ec8e90a252f9819d36bef8210df"
server_key_version: "encryptionkey@0"
I20260620 04:30:16.473407 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fc8b42c0376847348fdbf281a7a986f7"
format_stamp: "Formatted at 2026-06-20 04:30:16 on dist-test-slave-rzqm"
server_key: "00d27303ab1304179c3cce255a3aa457"
server_key_iv: "cb3b1ec8e90a252f9819d36bef8210df"
server_key_version: "encryptionkey@0"
I20260620 04:30:16.477586 2339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:16.480363 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:16.481798 2339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:16.481943 2339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "fc8b42c0376847348fdbf281a7a986f7"
format_stamp: "Formatted at 2026-06-20 04:30:16 on dist-test-slave-rzqm"
server_key: "00d27303ab1304179c3cce255a3aa457"
server_key_iv: "cb3b1ec8e90a252f9819d36bef8210df"
server_key_version: "encryptionkey@0"
I20260620 04:30:16.482071 2339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:16.508267 2339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:16.511556 2339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:16.511730 2339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:16.513530 2339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:16.513589 2339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:16.513695 2339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:16.513728 2339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:16.524385 2339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41093
I20260620 04:30:16.524438 2469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41093 every 8 connection(s)
I20260620 04:30:16.525399 2339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 04:30:16 dist-test-slave-rzqm krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:16.528486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 2339
I20260620 04:30:16.528597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:16.528846 392 external_mini_cluster.cc:1428] Setting key 2af8592981392e3db616e40f70108e7d
I20260620 04:30:16.531435 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.514340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53607 (local address 127.0.98.62:44581)
0620 04:30:16.514475 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.514480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.518433 (+ 3953us) server_negotiation.cc:407] Connection header received
0620 04:30:16.519758 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.519762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.519821 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.519876 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:16.521765 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.522580 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.524322 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.524641 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.524741 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.527477 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:16.527481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:16.527484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:16.527514 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:16.529100 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.529678 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.529681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.529683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.529774 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.530162 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.530165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.530167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.530393 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:16.530475 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.531206 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.531315 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:30:16.532509 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.514907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44581 (local address 127.0.98.2:53607)
0620 04:30:16.518190 (+ 3283us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.518255 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.519479 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.520295 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.520306 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.520840 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:16.521581 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.521606 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.522722 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.522752 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.524155 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.524167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.524803 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.524807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.525023 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.525632 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:16.525649 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:16.527234 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:16.529216 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.529223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.529239 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.529564 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.529870 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.529872 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.529873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.530050 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.530517 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:16.530526 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:16.530918 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.531137 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.531495 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2913,"thread_start_us":122,"threads_started":1}
I20260620 04:30:16.533993 2470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44581
I20260620 04:30:16.534152 2470 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:16.534750 2470 heartbeater.cc:511] Master 127.0.98.62:44581 requested a full tablet report, sending...
I20260620 04:30:16.535929 2127 ts_manager.cc:194] Registered new tserver with Master: fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093)
I20260620 04:30:16.536865 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53607
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:30:16.543540 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.536211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44581 (local address 127.0.98.2:35567)
0620 04:30:16.536483 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.536495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.536588 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.536792 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.536794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.536995 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:16.537184 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.537193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.538149 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.538152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.538997 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.539007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.539493 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.539498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.539573 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.539997 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:16.540007 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:16.540375 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:16.542024 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.542028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.542030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.542273 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.542552 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.542554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.542555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.542626 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.543068 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:16.543070 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:16.543147 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.543234 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.543402 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
I20260620 04:30:16.543543 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.536184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35567 (local address 127.0.98.62:44581)
0620 04:30:16.536294 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.536298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.536563 (+ 265us) server_negotiation.cc:407] Connection header received
0620 04:30:16.536661 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.536664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.536706 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.536766 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:16.537310 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.538005 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.539130 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.539341 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.539433 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.540520 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:16.540523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:16.540525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:16.540546 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:16.541909 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.542384 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.542387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.542389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.542474 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.542691 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.542693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.542695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.542972 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:16.543064 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.543314 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.543428 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:30:16.556205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44581
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:16.669265 2479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:16.669595 2479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:16.669644 2479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:16.673744 2479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:16.673805 2479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:16.673846 2479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 04:30:16.673859 2479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:30:16.673897 2479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:16.679209 2479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:16.680327 2479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:16.681406 2479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:16.690876 2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:16.690862 2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:16.690950 2479 server_base.cc:1034] running on GCE node
W20260620 04:30:16.690862 2485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:16.691497 2479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:16.692257 2479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:16.693456 2479 hybrid_clock.cc:648] HybridClock initialized: now 1781929816693426 us; error 40 us; skew 500 ppm
Jun 20 04:30:16 dist-test-slave-rzqm krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:16.696282 2479 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:30:16.697229 2479 webserver.cc:460] Webserver started at http://127.0.98.3:43249/ using document root <none> and password file <none>
I20260620 04:30:16.697762 2479 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:16.697822 2479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:16.698040 2479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:16.700048 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "efa2af3201e74f2583cd893cf5cc869e"
format_stamp: "Formatted at 2026-06-20 04:30:16 on dist-test-slave-rzqm"
server_key: "141b4db2bda8ec870f2d7217b309c458"
server_key_iv: "de72a47a3199912e9dccf0140802d4fe"
server_key_version: "encryptionkey@0"
I20260620 04:30:16.700582 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "efa2af3201e74f2583cd893cf5cc869e"
format_stamp: "Formatted at 2026-06-20 04:30:16 on dist-test-slave-rzqm"
server_key: "141b4db2bda8ec870f2d7217b309c458"
server_key_iv: "de72a47a3199912e9dccf0140802d4fe"
server_key_version: "encryptionkey@0"
I20260620 04:30:16.704291 2479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 04:30:16.706931 2494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:16.708211 2479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:16.708321 2479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "efa2af3201e74f2583cd893cf5cc869e"
format_stamp: "Formatted at 2026-06-20 04:30:16 on dist-test-slave-rzqm"
server_key: "141b4db2bda8ec870f2d7217b309c458"
server_key_iv: "de72a47a3199912e9dccf0140802d4fe"
server_key_version: "encryptionkey@0"
I20260620 04:30:16.708436 2479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:16.720469 2479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:16.723224 2479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:16.723407 2479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:16.724764 2479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:16.724814 2479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:16.724865 2479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:16.724895 2479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:30:16 dist-test-slave-rzqm krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:16.734651 2479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41729
I20260620 04:30:16.734741 2609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41729 every 8 connection(s)
I20260620 04:30:16.735579 2479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:16.737561 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.725525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46689 (local address 127.0.98.62:44581)
0620 04:30:16.725660 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.725665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.726502 (+ 837us) server_negotiation.cc:407] Connection header received
0620 04:30:16.727551 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.727554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.727609 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.727687 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:16.729288 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.729953 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.731138 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.731360 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.731447 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.733987 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:16.733990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:16.733992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:16.734025 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:16.735439 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.735946 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.735949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.735952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.736054 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.736392 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.736394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.736395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.736648 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:16.736736 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.737266 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.737377 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:30:16.738368 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.725923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44581 (local address 127.0.98.3:46689)
0620 04:30:16.726346 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.726393 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.727307 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.727918 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.727928 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.728393 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:16.729132 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.729155 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.730076 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.730080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.731016 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.731034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.731514 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.731518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.731681 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.732224 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:16.732240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:16.733814 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:16.735555 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.735561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.735577 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.735853 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.736162 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.736165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.736167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.736305 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.736741 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:16.736747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:16.736987 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.737218 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.737478 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260620 04:30:16.739660 2610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44581
I20260620 04:30:16.739809 2610 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:16.740360 2610 heartbeater.cc:511] Master 127.0.98.62:44581 requested a full tablet report, sending...
I20260620 04:30:16.741528 2126 ts_manager.cc:194] Registered new tserver with Master: efa2af3201e74f2583cd893cf5cc869e (127.0.98.3:41729)
I20260620 04:30:16.742218 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46689
I20260620 04:30:16.742703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 2479
I20260620 04:30:16.742817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:16.743157 392 external_mini_cluster.cc:1428] Setting key 3e3167989782c6ad2507583d9923ee72
I20260620 04:30:16.745119 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:30:16.748869 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.741731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39283 (local address 127.0.98.62:44581)
0620 04:30:16.741842 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.741847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.742050 (+ 203us) server_negotiation.cc:407] Connection header received
0620 04:30:16.742124 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.742127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.742170 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.742232 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:16.742724 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.743316 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.744322 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.744632 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.744727 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.745873 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:16.745877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:16.745880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:16.745905 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:16.747255 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.747744 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.747746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.747747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.747818 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.748080 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.748082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.748083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.748249 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:16.748317 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.748635 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.748732 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:30:16.748868 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.741672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44581 (local address 127.0.98.3:39283)
0620 04:30:16.741945 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.741957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.742050 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.742257 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.742259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.742473 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:16.742606 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.742615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.743440 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.743443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.744183 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.744193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.744740 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.744744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.744837 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.745386 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:16.745399 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:16.745739 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:16.747352 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.747356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.747358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.747655 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.747898 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:16.747900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:16.747902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:16.747995 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:16.748326 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:16.748329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:16.748424 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.748532 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.748671 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":108,"threads_started":1}
Jun 20 04:30:16 dist-test-slave-rzqm krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:30:16 dist-test-slave-rzqm krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:16.770875 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.763086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.0.98.62:44581)
0620 04:30:16.763207 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.763210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.763222 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:30:16.763260 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.763263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.763301 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.763375 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:16.764027 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.764529 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.765359 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.765597 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.765672 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.767891 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:16.767894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:16.767896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:16.767924 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:16.769264 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.769718 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.769721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.769722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.769807 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:16.770025 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:16.770027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:16.770028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:16.770202 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:16.770303 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.770558 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.770688 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:30:16.773916 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:16.776955 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:16.791137 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.785338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41729 (local address 127.0.0.1:56912)
0620 04:30:16.785673 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.785727 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.785876 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.786762 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.786768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.786791 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:16.786921 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.786927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.788218 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.788221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.789653 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.789663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.790605 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.790608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.790710 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.790747 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.790856 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.790985 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
I20260620 04:30:16.791150 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.785463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.0.98.3:41729)
0620 04:30:16.785775 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.785787 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.785810 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:30:16.786205 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.786211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.786426 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.786589 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:16.787058 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.788135 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.789775 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.790499 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.790596 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.790773 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.790896 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.790987 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":113,"threads_started":1}
I20260620 04:30:16.791558 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.786633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41093 (local address 127.0.0.1:36092)
0620 04:30:16.786885 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.786898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.786983 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.787770 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.787773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.787788 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:16.787887 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.787894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.789279 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.789283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.790173 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.790180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.791204 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.791207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.791292 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.791313 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.791362 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.791429 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
I20260620 04:30:16.791731 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.786657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40331 (local address 127.0.0.1:45586)
0620 04:30:16.787395 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.787406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.787481 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.787959 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.787962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.787977 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:16.788071 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.788077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.789126 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.789130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.790130 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.790140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.791387 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.791390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.791485 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.791504 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.791566 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.791626 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":102,"threads_started":1}
I20260620 04:30:16.792166 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.787068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.98.1:40331)
0620 04:30:16.787446 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.787461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.787484 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:30:16.787655 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.787661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.787844 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.787983 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:16.788170 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.788984 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.790258 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.791074 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.791177 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.791334 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.791916 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.792017 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":80,"threads_started":1}
I20260620 04:30:16.792236 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.787058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.0.98.2:41093)
0620 04:30:16.787336 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.787346 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.787368 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:30:16.787443 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.787448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.787658 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.787788 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:16.788012 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.789161 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.790308 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.791074 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.791809 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.791988 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.792032 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.792114 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
I20260620 04:30:16.794229 2544 tablet_service.cc:1467] Processing CreateTablet for tablet cc553f37992e4f17a6a1fc49751e7b3e (DEFAULT_TABLE table=test-table [id=29b4cf9856f84e849d3f43f4b09a9f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:16.795185 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc553f37992e4f17a6a1fc49751e7b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:16.794847 2403 tablet_service.cc:1467] Processing CreateTablet for tablet cc553f37992e4f17a6a1fc49751e7b3e (DEFAULT_TABLE table=test-table [id=29b4cf9856f84e849d3f43f4b09a9f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:16.794848 2264 tablet_service.cc:1467] Processing CreateTablet for tablet cc553f37992e4f17a6a1fc49751e7b3e (DEFAULT_TABLE table=test-table [id=29b4cf9856f84e849d3f43f4b09a9f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:16.795832 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc553f37992e4f17a6a1fc49751e7b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:16.795832 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc553f37992e4f17a6a1fc49751e7b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:16.802302 2633 tablet_bootstrap.cc:492] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: Bootstrap starting.
I20260620 04:30:16.803194 2634 tablet_bootstrap.cc:492] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: Bootstrap starting.
I20260620 04:30:16.804339 2635 tablet_bootstrap.cc:492] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: Bootstrap starting.
I20260620 04:30:16.804832 2633 tablet_bootstrap.cc:654] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:16.805392 2634 tablet_bootstrap.cc:654] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:16.805769 2633 log.cc:824] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:16.806064 2635 tablet_bootstrap.cc:654] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:16.806159 2634 log.cc:824] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:16.806772 2635 log.cc:824] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:16.807790 2633 tablet_bootstrap.cc:492] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: No bootstrap required, opened a new log
I20260620 04:30:16.807966 2633 ts_tablet_manager.cc:1397] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:30:16.808090 2634 tablet_bootstrap.cc:492] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: No bootstrap required, opened a new log
I20260620 04:30:16.808274 2634 ts_tablet_manager.cc:1397] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:30:16.808532 2635 tablet_bootstrap.cc:492] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: No bootstrap required, opened a new log
I20260620 04:30:16.808691 2635 ts_tablet_manager.cc:1397] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 04:30:16.810904 2633 raft_consensus.cc:348] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } }
I20260620 04:30:16.811125 2633 raft_consensus.cc:374] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:16.811175 2633 raft_consensus.cc:729] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa2af3201e74f2583cd893cf5cc869e, State: Initialized, Role: FOLLOWER
I20260620 04:30:16.811568 2633 consensus_queue.cc:260] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } }
I20260620 04:30:16.812265 2610 heartbeater.cc:503] Master 127.0.98.62:44581 was elected leader, sending a full tablet report...
I20260620 04:30:16.812156 2634 raft_consensus.cc:348] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } }
I20260620 04:30:16.812156 2635 raft_consensus.cc:348] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } }
I20260620 04:30:16.812387 2635 raft_consensus.cc:374] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:16.812387 2634 raft_consensus.cc:374] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:16.812433 2634 raft_consensus.cc:729] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8b42c0376847348fdbf281a7a986f7, State: Initialized, Role: FOLLOWER
I20260620 04:30:16.812433 2635 raft_consensus.cc:729] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062b3f3e9c124b8a9e25c830f95de006, State: Initialized, Role: FOLLOWER
I20260620 04:30:16.812507 2633 ts_tablet_manager.cc:1428] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 04:30:16.812853 2635 consensus_queue.cc:260] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } }
I20260620 04:30:16.812831 2634 consensus_queue.cc:260] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } }
I20260620 04:30:16.813478 2330 heartbeater.cc:503] Master 127.0.98.62:44581 was elected leader, sending a full tablet report...
I20260620 04:30:16.813838 2470 heartbeater.cc:503] Master 127.0.98.62:44581 was elected leader, sending a full tablet report...
I20260620 04:30:16.814014 2634 ts_tablet_manager.cc:1428] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:30:16.814893 2635 ts_tablet_manager.cc:1428] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:30:16.948637 2639 raft_consensus.cc:482] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:16.948853 2639 raft_consensus.cc:504] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } }
I20260620 04:30:16.950062 2639 leader_election.cc:288] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331), fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093)
I20260620 04:30:16.954159 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.950383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58833 (local address 127.0.98.1:40331)
0620 04:30:16.950501 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.950506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.950580 (+ 74us) server_negotiation.cc:407] Connection header received
0620 04:30:16.950726 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.950729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.950785 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.950867 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:16.951152 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.951779 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.952946 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.953574 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.953688 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.953751 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.953987 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.954050 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:30:16.954198 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.950319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40331 (local address 127.0.98.3:58833)
0620 04:30:16.950479 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.950496 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.950607 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.950894 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.950896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.950913 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:16.951041 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.951049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.951887 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.951890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.952799 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.952809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.953736 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.953740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.953854 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.953881 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.953965 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.954056 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260620 04:30:16.954826 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.950335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41093 (local address 127.0.98.3:57711)
0620 04:30:16.950885 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:16.950900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:16.950995 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:16.951221 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:16.951223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:16.951239 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:16.951358 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.951364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.952133 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.952135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.953574 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:16.953583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.954447 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.954451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:16.954576 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.954596 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:30:16.954659 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:30:16.954713 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":202,"thread_start_us":112,"threads_started":1}
I20260620 04:30:16.954913 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc553f37992e4f17a6a1fc49751e7b3e" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062b3f3e9c124b8a9e25c830f95de006" is_pre_election: true
I20260620 04:30:16.955258 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:16.950443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57711 (local address 127.0.98.2:41093)
0620 04:30:16.950555 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:16.950560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:16.950980 (+ 420us) server_negotiation.cc:407] Connection header received
0620 04:30:16.951083 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:16.951086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:16.951135 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:16.951195 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:16.951449 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.952030 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.953716 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:16.954313 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:16.954414 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:16.954475 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:16.955099 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 04:30:16.955170 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 04:30:16.955258 2284 raft_consensus.cc:2454] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 0.
I20260620 04:30:16.955787 2498 leader_election.cc:302] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 062b3f3e9c124b8a9e25c830f95de006, efa2af3201e74f2583cd893cf5cc869e; no voters:
I20260620 04:30:16.955821 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc553f37992e4f17a6a1fc49751e7b3e" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8b42c0376847348fdbf281a7a986f7" is_pre_election: true
I20260620 04:30:16.956084 2639 raft_consensus.cc:2781] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:16.956096 2413 raft_consensus.cc:2454] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 0.
I20260620 04:30:16.956167 2639 raft_consensus.cc:482] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:16.956218 2639 raft_consensus.cc:3037] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:16.957293 2639 raft_consensus.cc:504] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } }
I20260620 04:30:16.957628 2639 leader_election.cc:288] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 election: Requested vote from peers 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331), fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093)
I20260620 04:30:16.957950 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc553f37992e4f17a6a1fc49751e7b3e" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8b42c0376847348fdbf281a7a986f7"
I20260620 04:30:16.958009 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc553f37992e4f17a6a1fc49751e7b3e" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062b3f3e9c124b8a9e25c830f95de006"
I20260620 04:30:16.958081 2413 raft_consensus.cc:3037] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:16.958119 2284 raft_consensus.cc:3037] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:16.959077 2284 raft_consensus.cc:2454] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 1.
I20260620 04:30:16.959089 2413 raft_consensus.cc:2454] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 1.
I20260620 04:30:16.959339 2498 leader_election.cc:302] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 062b3f3e9c124b8a9e25c830f95de006, efa2af3201e74f2583cd893cf5cc869e; no voters:
I20260620 04:30:16.959571 2639 raft_consensus.cc:2781] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:16.959749 2639 raft_consensus.cc:686] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 1 LEADER]: Becoming Leader. State: Replica: efa2af3201e74f2583cd893cf5cc869e, State: Running, Role: LEADER
I20260620 04:30:16.960044 2639 consensus_queue.cc:237] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } }
I20260620 04:30:16.963507 2125 catalog_manager.cc:5495] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e reported cstate change: term changed from 0 to 1, leader changed from <none> to efa2af3201e74f2583cd893cf5cc869e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "efa2af3201e74f2583cd893cf5cc869e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } health_report { overall_health: UNKNOWN } } }
W20260620 04:30:16.986577 2611 tablet.cc:2369] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 04:30:17 dist-test-slave-rzqm krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:17.018502 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.010721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44581 (local address 127.0.0.1:49686)
0620 04:30:17.011004 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:17.011016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:17.011142 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:17.011356 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:17.011358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:17.011751 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:17.011845 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.011851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.012585 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.012587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.013208 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.013217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.013623 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.013627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.013712 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.014235 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:17.014245 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:17.015560 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:17.016996 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:17.017043 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:17.017051 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:17.017301 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:17.017550 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:17.017553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:17.017554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:17.017635 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:17.017957 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:17.017963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:17.018072 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 04:30:17.018161 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 04:30:17.018275 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":125,"threads_started":1}
I20260620 04:30:17.018523 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.010780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.0.98.62:44581)
0620 04:30:17.010951 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:17.010956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:17.011115 (+ 159us) server_negotiation.cc:407] Connection header received
0620 04:30:17.011219 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:17.011222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:17.011263 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:17.011330 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:17.011943 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.012495 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.013311 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.013515 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.013598 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.015682 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:17.015685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:17.015686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:17.015707 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:17.016878 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:17.017390 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:17.017393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:17.017395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:17.017468 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:17.017702 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:17.017705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:17.017706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:17.017888 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:17.017953 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:17.018250 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 04:30:17.018364 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 04:30:17.022482 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49686:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 04:30:17.023243 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260620 04:30:17.026584 2471 tablet.cc:2369] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:30:17.028007 2264 tablet_service.cc:1467] Processing CreateTablet for tablet c702273d0d3f4c588f03fcb4848016cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64ab0f7b86bc43228d3159e247f0a50d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:30:17.028169 2544 tablet_service.cc:1467] Processing CreateTablet for tablet c702273d0d3f4c588f03fcb4848016cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64ab0f7b86bc43228d3159e247f0a50d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:30:17.028736 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c702273d0d3f4c588f03fcb4848016cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:17.028826 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c702273d0d3f4c588f03fcb4848016cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:17.031155 2633 tablet_bootstrap.cc:492] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: Bootstrap starting.
I20260620 04:30:17.031977 2633 tablet_bootstrap.cc:654] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:17.033100 2633 tablet_bootstrap.cc:492] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: No bootstrap required, opened a new log
I20260620 04:30:17.033193 2633 ts_tablet_manager.cc:1397] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:17.032883 2403 tablet_service.cc:1467] Processing CreateTablet for tablet c702273d0d3f4c588f03fcb4848016cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64ab0f7b86bc43228d3159e247f0a50d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:30:17.033629 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c702273d0d3f4c588f03fcb4848016cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:17.033608 2633 raft_consensus.cc:348] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.033716 2633 raft_consensus.cc:374] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:17.033754 2633 raft_consensus.cc:729] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa2af3201e74f2583cd893cf5cc869e, State: Initialized, Role: FOLLOWER
I20260620 04:30:17.033878 2633 consensus_queue.cc:260] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.034139 2633 ts_tablet_manager.cc:1428] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:17.034324 2639 tablet_replica.cc:439] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } }
I20260620 04:30:17.034430 2639 tablet_replica.cc:442] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:17.036196 2634 tablet_bootstrap.cc:492] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: Bootstrap starting.
I20260620 04:30:17.036690 2635 tablet_bootstrap.cc:492] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: Bootstrap starting.
I20260620 04:30:17.037182 2634 tablet_bootstrap.cc:654] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:17.037627 2635 tablet_bootstrap.cc:654] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:17.038618 2634 tablet_bootstrap.cc:492] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: No bootstrap required, opened a new log
I20260620 04:30:17.038710 2634 ts_tablet_manager.cc:1397] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:17.038753 2635 tablet_bootstrap.cc:492] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: No bootstrap required, opened a new log
I20260620 04:30:17.038834 2635 ts_tablet_manager.cc:1397] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:17.039187 2634 raft_consensus.cc:348] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.039310 2634 raft_consensus.cc:374] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:17.039274 2635 raft_consensus.cc:348] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.039348 2634 raft_consensus.cc:729] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8b42c0376847348fdbf281a7a986f7, State: Initialized, Role: FOLLOWER
I20260620 04:30:17.039363 2635 raft_consensus.cc:374] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:17.039393 2635 raft_consensus.cc:729] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062b3f3e9c124b8a9e25c830f95de006, State: Initialized, Role: FOLLOWER
I20260620 04:30:17.039480 2634 consensus_queue.cc:260] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.039499 2635 consensus_queue.cc:260] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.039748 2635 ts_tablet_manager.cc:1428] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:17.039750 2640 tablet_replica.cc:439] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } }
I20260620 04:30:17.039914 2640 tablet_replica.cc:442] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:17.040167 2641 tablet_replica.cc:439] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } }
I20260620 04:30:17.040325 2641 tablet_replica.cc:442] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:17.040159 2634 ts_tablet_manager.cc:1428] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260620 04:30:17.051023 2331 tablet.cc:2369] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:30:17.182515 2639 raft_consensus.cc:482] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:17.182662 2639 raft_consensus.cc:504] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.183084 2639 leader_election.cc:288] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331), fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093)
I20260620 04:30:17.183595 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702273d0d3f4c588f03fcb4848016cb" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062b3f3e9c124b8a9e25c830f95de006" is_pre_election: true
I20260620 04:30:17.183596 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702273d0d3f4c588f03fcb4848016cb" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8b42c0376847348fdbf281a7a986f7" is_pre_election: true
I20260620 04:30:17.183758 2413 raft_consensus.cc:2454] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 0.
I20260620 04:30:17.183758 2284 raft_consensus.cc:2454] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 0.
I20260620 04:30:17.184140 2497 leader_election.cc:302] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efa2af3201e74f2583cd893cf5cc869e, fc8b42c0376847348fdbf281a7a986f7; no voters:
I20260620 04:30:17.184324 2639 raft_consensus.cc:2781] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:17.184391 2639 raft_consensus.cc:482] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:17.184418 2639 raft_consensus.cc:3037] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:17.185540 2639 raft_consensus.cc:504] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.186034 2639 leader_election.cc:288] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 election: Requested vote from peers 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331), fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093)
I20260620 04:30:17.186333 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702273d0d3f4c588f03fcb4848016cb" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062b3f3e9c124b8a9e25c830f95de006"
I20260620 04:30:17.186422 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702273d0d3f4c588f03fcb4848016cb" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8b42c0376847348fdbf281a7a986f7"
I20260620 04:30:17.186482 2284 raft_consensus.cc:3037] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:17.186543 2413 raft_consensus.cc:3037] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:17.187666 2284 raft_consensus.cc:2454] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 1.
I20260620 04:30:17.187680 2413 raft_consensus.cc:2454] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 1.
I20260620 04:30:17.187966 2498 leader_election.cc:302] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 062b3f3e9c124b8a9e25c830f95de006, efa2af3201e74f2583cd893cf5cc869e; no voters:
I20260620 04:30:17.188136 2639 raft_consensus.cc:2781] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:17.188223 2639 raft_consensus.cc:686] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [term 1 LEADER]: Becoming Leader. State: Replica: efa2af3201e74f2583cd893cf5cc869e, State: Running, Role: LEADER
I20260620 04:30:17.188359 2639 consensus_queue.cc:237] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.189069 2643 tablet_replica.cc:439] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: TxnStatusTablet state changed. Reason: New leader efa2af3201e74f2583cd893cf5cc869e. Latest consensus state: current_term: 1 leader_uuid: "efa2af3201e74f2583cd893cf5cc869e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } }
I20260620 04:30:17.189167 2643 tablet_replica.cc:442] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:30:17.189456 2655 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 04:30:17.189672 2655 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 04:30:17.190140 2125 catalog_manager.cc:5495] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e reported cstate change: term changed from 0 to 1, leader changed from <none> to efa2af3201e74f2583cd893cf5cc869e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "efa2af3201e74f2583cd893cf5cc869e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } health_report { overall_health: HEALTHY } } }
I20260620 04:30:17.226428 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.222313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.0.98.3:41729)
0620 04:30:17.222423 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:17.222427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:17.222536 (+ 109us) server_negotiation.cc:407] Connection header received
0620 04:30:17.222737 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:17.222739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:17.222793 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:17.222874 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:17.223113 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.223646 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.224866 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.225175 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.225258 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.225606 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:17.226061 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:17.226150 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:17.226255 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 04:30:17.226311 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:30:17.226459 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.222221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41729 (local address 127.0.0.1:56926)
0620 04:30:17.222465 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:17.222480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:17.222607 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:17.222903 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:17.222906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:17.222914 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:17.223023 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.223028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.223746 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.223749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.224729 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.224740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.225281 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.225285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.225374 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.225430 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:17.226154 (+ 724us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:17.226168 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 04:30:17.226238 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:30:17.226326 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260620 04:30:17.232249 2284 raft_consensus.cc:1261] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Refusing update from remote peer efa2af3201e74f2583cd893cf5cc869e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:17.232249 2413 raft_consensus.cc:1261] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Refusing update from remote peer efa2af3201e74f2583cd893cf5cc869e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:17.233093 2639 consensus_queue.cc:1035] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [LEADER]: Connected to new peer: Peer: permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:17.233281 2643 consensus_queue.cc:1035] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:17.237774 2641 tablet_replica.cc:439] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: TxnStatusTablet state changed. Reason: New leader efa2af3201e74f2583cd893cf5cc869e. Latest consensus state: current_term: 1 leader_uuid: "efa2af3201e74f2583cd893cf5cc869e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } }
I20260620 04:30:17.237774 2640 tablet_replica.cc:439] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: TxnStatusTablet state changed. Reason: New leader efa2af3201e74f2583cd893cf5cc869e. Latest consensus state: current_term: 1 leader_uuid: "efa2af3201e74f2583cd893cf5cc869e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } }
I20260620 04:30:17.237927 2640 tablet_replica.cc:442] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:17.237927 2641 tablet_replica.cc:442] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:17.241532 2639 tablet_replica.cc:439] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efa2af3201e74f2583cd893cf5cc869e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } }
I20260620 04:30:17.241686 2639 tablet_replica.cc:442] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:30:17.241806 2639 tablet_replica.cc:439] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efa2af3201e74f2583cd893cf5cc869e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } }
I20260620 04:30:17.241878 2639 tablet_replica.cc:442] T c702273d0d3f4c588f03fcb4848016cb P efa2af3201e74f2583cd893cf5cc869e: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:30:17.241896 2641 tablet_replica.cc:439] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efa2af3201e74f2583cd893cf5cc869e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } }
I20260620 04:30:17.242017 2641 tablet_replica.cc:442] T c702273d0d3f4c588f03fcb4848016cb P fc8b42c0376847348fdbf281a7a986f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:17.242594 2640 tablet_replica.cc:439] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efa2af3201e74f2583cd893cf5cc869e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } } }
I20260620 04:30:17.242707 2640 tablet_replica.cc:442] T c702273d0d3f4c588f03fcb4848016cb P 062b3f3e9c124b8a9e25c830f95de006: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:17.243225 2658 mvcc.cc:205] Tried to move back new op lower bound from 7298784531375837184 to 7298784531204624384. Current Snapshot: MvccSnapshot[applied={T|T < 7298784531375837184}]
I20260620 04:30:17.243681 2659 mvcc.cc:205] Tried to move back new op lower bound from 7298784531375837184 to 7298784531204624384. Current Snapshot: MvccSnapshot[applied={T|T < 7298784531375837184}]
I20260620 04:30:17.243834 2660 mvcc.cc:205] Tried to move back new op lower bound from 7298784531375837184 to 7298784531204624384. Current Snapshot: MvccSnapshot[applied={T|T < 7298784531375837184}]
I20260620 04:30:17.251710 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.248822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.0.98.3:41729)
0620 04:30:17.248920 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:17.248924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:17.248937 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:17.249091 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:17.249094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:17.249139 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:17.249199 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:17.249449 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.249941 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.250781 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.250996 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.251077 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.251320 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:17.251392 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:17.251449 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:17.251530 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 04:30:17.251582 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:30:17.262372 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.258934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39063 (local address 127.0.98.3:41729)
0620 04:30:17.259050 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:17.259054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:17.259303 (+ 249us) server_negotiation.cc:407] Connection header received
0620 04:30:17.259410 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:17.259413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:17.259453 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:17.259511 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:17.259761 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.260269 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.261225 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.261476 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.261559 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.261876 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:17.261973 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:17.262033 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:17.262121 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 04:30:17.262194 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":45}
I20260620 04:30:17.262382 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.258866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41729 (local address 127.0.98.3:39063)
0620 04:30:17.259227 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:17.259238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:17.259338 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:17.259527 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:17.259530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:17.259536 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:17.259673 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.259680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.260359 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.260362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.261129 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.261138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.261592 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.261595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.261695 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.261734 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:17.262036 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:17.262042 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:30:17.262112 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:30:17.262206 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":20,"thread_start_us":96,"threads_started":1}
I20260620 04:30:17.270532 2413 raft_consensus.cc:1261] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Refusing update from remote peer efa2af3201e74f2583cd893cf5cc869e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:17.270931 2284 raft_consensus.cc:1261] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Refusing update from remote peer efa2af3201e74f2583cd893cf5cc869e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:17.271001 2639 consensus_queue.cc:1035] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:17.271281 2643 consensus_queue.cc:1035] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [LEADER]: Connected to new peer: Peer: permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:17.326905 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:17.327137 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:17.329967 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78: Sending DeleteTablet for 3 replicas of tablet cc553f37992e4f17a6a1fc49751e7b3e
I20260620 04:30:17.330833 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet cc553f37992e4f17a6a1fc49751e7b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45586
I20260620 04:30:17.330833 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet cc553f37992e4f17a6a1fc49751e7b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56912
I20260620 04:30:17.331331 2675 tablet_replica.cc:330] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: stopping tablet replica
I20260620 04:30:17.331512 2676 tablet_replica.cc:330] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: stopping tablet replica
I20260620 04:30:17.331540 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:30:17.331722 2675 raft_consensus.cc:2229] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:17.331990 2675 raft_consensus.cc:2258] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 04:30:17.331995 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:17.332048 2676 raft_consensus.cc:2229] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:17.332556 2676 raft_consensus.cc:2258] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:17.333319 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet cc553f37992e4f17a6a1fc49751e7b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36092
I20260620 04:30:17.333547 2675 ts_tablet_manager.cc:1905] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:17.336359 2676 ts_tablet_manager.cc:1905] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:17.337031 2675 ts_tablet_manager.cc:1918] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:17.337142 2675 log.cc:1196] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/cc553f37992e4f17a6a1fc49751e7b3e
I20260620 04:30:17.337446 2675 ts_tablet_manager.cc:1939] T cc553f37992e4f17a6a1fc49751e7b3e P 062b3f3e9c124b8a9e25c830f95de006: Deleting consensus metadata
I20260620 04:30:17.337504 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 1f281f365d9a4c0aabfeb2a456d6ed4b (DEFAULT_TABLE table=test-table [id=71fbe38d855b405b84dd1d4c4bb2bfd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:17.337738 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f281f365d9a4c0aabfeb2a456d6ed4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:17.337776 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 1f281f365d9a4c0aabfeb2a456d6ed4b (DEFAULT_TABLE table=test-table [id=71fbe38d855b405b84dd1d4c4bb2bfd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:17.338013 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f281f365d9a4c0aabfeb2a456d6ed4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:17.338927 2114 catalog_manager.cc:4841] TS 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331): tablet cc553f37992e4f17a6a1fc49751e7b3e (table test-table [id=29b4cf9856f84e849d3f43f4b09a9f25]) successfully deleted
I20260620 04:30:17.340270 2635 tablet_bootstrap.cc:492] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006: Bootstrap starting.
I20260620 04:30:17.340507 2633 tablet_bootstrap.cc:492] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e: Bootstrap starting.
I20260620 04:30:17.341269 2635 tablet_bootstrap.cc:654] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:17.341437 2633 tablet_bootstrap.cc:654] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:17.341805 2676 ts_tablet_manager.cc:1918] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:17.341917 2676 log.cc:1196] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/cc553f37992e4f17a6a1fc49751e7b3e
I20260620 04:30:17.342231 2676 ts_tablet_manager.cc:1939] T cc553f37992e4f17a6a1fc49751e7b3e P efa2af3201e74f2583cd893cf5cc869e: Deleting consensus metadata
I20260620 04:30:17.342964 2633 tablet_bootstrap.cc:492] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e: No bootstrap required, opened a new log
I20260620 04:30:17.343070 2633 ts_tablet_manager.cc:1397] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:17.343492 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 1f281f365d9a4c0aabfeb2a456d6ed4b (DEFAULT_TABLE table=test-table [id=71fbe38d855b405b84dd1d4c4bb2bfd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:17.343532 2633 raft_consensus.cc:348] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.343647 2633 raft_consensus.cc:374] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:17.343676 2633 raft_consensus.cc:729] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa2af3201e74f2583cd893cf5cc869e, State: Initialized, Role: FOLLOWER
I20260620 04:30:17.343772 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f281f365d9a4c0aabfeb2a456d6ed4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:17.343789 2633 consensus_queue.cc:260] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.344038 2633 ts_tablet_manager.cc:1428] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:17.344421 2677 tablet_replica.cc:330] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: stopping tablet replica
I20260620 04:30:17.344563 2635 tablet_bootstrap.cc:492] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006: No bootstrap required, opened a new log
I20260620 04:30:17.344650 2635 ts_tablet_manager.cc:1397] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 04:30:17.344916 2677 raft_consensus.cc:2229] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:17.345118 2635 raft_consensus.cc:348] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.345235 2635 raft_consensus.cc:374] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:17.345271 2635 raft_consensus.cc:729] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062b3f3e9c124b8a9e25c830f95de006, State: Initialized, Role: FOLLOWER
I20260620 04:30:17.345216 2677 raft_consensus.cc:2258] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:17.345389 2635 consensus_queue.cc:260] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.345652 2635 ts_tablet_manager.cc:1428] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:17.346160 2114 catalog_manager.cc:4841] TS efa2af3201e74f2583cd893cf5cc869e (127.0.98.3:41729): tablet cc553f37992e4f17a6a1fc49751e7b3e (table test-table [id=29b4cf9856f84e849d3f43f4b09a9f25]) successfully deleted
I20260620 04:30:17.346220 2634 tablet_bootstrap.cc:492] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7: Bootstrap starting.
I20260620 04:30:17.347075 2634 tablet_bootstrap.cc:654] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:17.347164 2677 ts_tablet_manager.cc:1905] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:17.349108 2634 tablet_bootstrap.cc:492] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7: No bootstrap required, opened a new log
I20260620 04:30:17.349198 2634 ts_tablet_manager.cc:1397] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:17.349628 2634 raft_consensus.cc:348] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.349742 2634 raft_consensus.cc:374] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:17.349772 2634 raft_consensus.cc:729] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8b42c0376847348fdbf281a7a986f7, State: Initialized, Role: FOLLOWER
I20260620 04:30:17.349889 2634 consensus_queue.cc:260] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.350162 2634 ts_tablet_manager.cc:1428] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:17.350288 2677 ts_tablet_manager.cc:1918] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:17.350378 2677 log.cc:1196] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/cc553f37992e4f17a6a1fc49751e7b3e
I20260620 04:30:17.350698 2677 ts_tablet_manager.cc:1939] T cc553f37992e4f17a6a1fc49751e7b3e P fc8b42c0376847348fdbf281a7a986f7: Deleting consensus metadata
I20260620 04:30:17.352026 2113 catalog_manager.cc:4841] TS fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093): tablet cc553f37992e4f17a6a1fc49751e7b3e (table test-table [id=29b4cf9856f84e849d3f43f4b09a9f25]) successfully deleted
I20260620 04:30:17.416563 2640 raft_consensus.cc:482] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:17.416735 2640 raft_consensus.cc:504] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.418047 2640 leader_election.cc:288] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093), efa2af3201e74f2583cd893cf5cc869e (127.0.98.3:41729)
I20260620 04:30:17.423334 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.418429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49857 (local address 127.0.98.2:41093)
0620 04:30:17.418601 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:17.418606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:17.418992 (+ 386us) server_negotiation.cc:407] Connection header received
0620 04:30:17.419189 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:17.419193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:17.419244 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:17.419311 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:17.419555 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.420180 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.422077 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.422711 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.422835 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.422904 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:17.423143 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 04:30:17.423217 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 04:30:17.423359 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.418303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41093 (local address 127.0.98.1:49857)
0620 04:30:17.418902 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:17.418915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:17.419025 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:17.419358 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:17.419360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:17.419372 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:17.419463 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.419469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.420293 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.420297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.421935 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.421948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.422905 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.422908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.422991 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.423013 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:30:17.423103 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 04:30:17.423186 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":463,"thread_start_us":86,"threads_started":1}
I20260620 04:30:17.424139 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.418506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35175 (local address 127.0.98.3:41729)
0620 04:30:17.418635 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:17.418640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:17.418871 (+ 231us) server_negotiation.cc:407] Connection header received
0620 04:30:17.419108 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:17.419112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:17.419170 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:17.419238 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:17.419486 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.420279 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.422971 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.423582 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.423684 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.423744 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:17.423907 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 04:30:17.424010 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:30:17.424269 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f281f365d9a4c0aabfeb2a456d6ed4b" candidate_uuid: "062b3f3e9c124b8a9e25c830f95de006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8b42c0376847348fdbf281a7a986f7" is_pre_election: true
I20260620 04:30:17.424441 2424 raft_consensus.cc:2454] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 062b3f3e9c124b8a9e25c830f95de006 in term 0.
I20260620 04:30:17.424559 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.418303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41729 (local address 127.0.98.1:35175)
0620 04:30:17.418771 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:17.418786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:17.418967 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:17.419253 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:17.419256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:17.419271 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:17.419373 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.419380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.420411 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.420414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.421935 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.421948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.423696 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.423700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.423769 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.423788 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:30:17.424338 (+ 550us) client_negotiation.cc:240] Negotiation successful
0620 04:30:17.424420 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":91,"thread_start_us":127,"threads_started":1}
I20260620 04:30:17.424899 2217 leader_election.cc:302] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 062b3f3e9c124b8a9e25c830f95de006, fc8b42c0376847348fdbf281a7a986f7; no voters:
I20260620 04:30:17.425079 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f281f365d9a4c0aabfeb2a456d6ed4b" candidate_uuid: "062b3f3e9c124b8a9e25c830f95de006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa2af3201e74f2583cd893cf5cc869e" is_pre_election: true
I20260620 04:30:17.425228 2640 raft_consensus.cc:2781] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:17.425275 2640 raft_consensus.cc:482] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:17.425307 2640 raft_consensus.cc:3037] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:17.425303 2564 raft_consensus.cc:2454] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 062b3f3e9c124b8a9e25c830f95de006 in term 0.
I20260620 04:30:17.426538 2640 raft_consensus.cc:504] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.426908 2640 leader_election.cc:288] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [CANDIDATE]: Term 1 election: Requested vote from peers fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093), efa2af3201e74f2583cd893cf5cc869e (127.0.98.3:41729)
I20260620 04:30:17.427274 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f281f365d9a4c0aabfeb2a456d6ed4b" candidate_uuid: "062b3f3e9c124b8a9e25c830f95de006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8b42c0376847348fdbf281a7a986f7"
I20260620 04:30:17.427392 2424 raft_consensus.cc:3037] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:17.428380 2424 raft_consensus.cc:2454] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 062b3f3e9c124b8a9e25c830f95de006 in term 1.
I20260620 04:30:17.428634 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f281f365d9a4c0aabfeb2a456d6ed4b" candidate_uuid: "062b3f3e9c124b8a9e25c830f95de006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa2af3201e74f2583cd893cf5cc869e"
I20260620 04:30:17.428674 2217 leader_election.cc:302] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 062b3f3e9c124b8a9e25c830f95de006, fc8b42c0376847348fdbf281a7a986f7; no voters:
I20260620 04:30:17.428762 2564 raft_consensus.cc:3037] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:17.428845 2640 raft_consensus.cc:2781] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:17.429131 2640 raft_consensus.cc:686] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 1 LEADER]: Becoming Leader. State: Replica: 062b3f3e9c124b8a9e25c830f95de006, State: Running, Role: LEADER
I20260620 04:30:17.429391 2640 consensus_queue.cc:237] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:17.429764 2564 raft_consensus.cc:2454] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 062b3f3e9c124b8a9e25c830f95de006 in term 1.
I20260620 04:30:17.431825 2123 catalog_manager.cc:5495] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 reported cstate change: term changed from 0 to 1, leader changed from <none> to 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "062b3f3e9c124b8a9e25c830f95de006" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } health_report { overall_health: UNKNOWN } } }
I20260620 04:30:17.458324 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.454250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.0.98.1:40331)
0620 04:30:17.454366 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:17.454370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:17.454386 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:30:17.454606 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:17.454610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:17.454677 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:17.454797 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:17.455052 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.455787 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.456639 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.456970 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.457092 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.457347 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:17.457952 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:17.458074 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:17.458105 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 04:30:17.458164 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:30:17.469424 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.465876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41729 (local address 127.0.98.1:46237)
0620 04:30:17.466142 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:17.466155 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:17.466245 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:17.466472 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:17.466474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:17.466479 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:17.466611 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.466616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.467586 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.467589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.468199 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.468207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.468741 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.468744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.468814 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.468857 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:17.469173 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:17.469178 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:30:17.469223 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 04:30:17.469286 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260620 04:30:17.469421 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.465990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46237 (local address 127.0.98.3:41729)
0620 04:30:17.466123 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:17.466128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:17.466223 (+ 95us) server_negotiation.cc:407] Connection header received
0620 04:30:17.466323 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:17.466327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:17.466380 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:17.466460 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:17.466711 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.467478 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.468320 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.468633 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.468740 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.468988 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:17.469097 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:17.469177 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:17.469240 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 04:30:17.469289 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:30:17.476002 2424 raft_consensus.cc:1261] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Refusing update from remote peer 062b3f3e9c124b8a9e25c830f95de006: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:17.476318 2564 raft_consensus.cc:1261] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Refusing update from remote peer 062b3f3e9c124b8a9e25c830f95de006: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:17.476509 2640 consensus_queue.cc:1035] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:17.476837 2683 consensus_queue.cc:1035] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:17.495445 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.491935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40331 (local address 127.0.98.3:60389)
0620 04:30:17.492140 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:17.492155 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:17.492218 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:17.492440 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:17.492443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:17.492450 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:17.492556 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.492561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.493512 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.493516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.494194 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:17.494204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.494748 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.494752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:17.494838 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.494873 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:17.495151 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:17.495157 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:30:17.495229 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 04:30:17.495320 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 04:30:17.495440 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:17.492154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60389 (local address 127.0.98.1:40331)
0620 04:30:17.492254 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:17.492258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:17.492271 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:17.492312 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:17.492315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:17.492360 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:17.492431 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:17.492669 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.493391 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.494313 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:17.494648 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:17.494747 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:17.494999 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:17.495070 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:17.495127 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:17.495248 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 04:30:17.495306 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 04:30:17.496170 2659 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 04:30:19.060588 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:19.060750 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:19.063691 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78: Sending DeleteTablet for 3 replicas of tablet 1f281f365d9a4c0aabfeb2a456d6ed4b
I20260620 04:30:19.064306 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f281f365d9a4c0aabfeb2a456d6ed4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45586
I20260620 04:30:19.064304 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f281f365d9a4c0aabfeb2a456d6ed4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36092
I20260620 04:30:19.064345 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f281f365d9a4c0aabfeb2a456d6ed4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56912
I20260620 04:30:19.064762 2717 tablet_replica.cc:330] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e: stopping tablet replica
I20260620 04:30:19.064878 2717 raft_consensus.cc:2229] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:19.064980 2717 raft_consensus.cc:2258] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:19.065205 2716 tablet_replica.cc:330] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7: stopping tablet replica
I20260620 04:30:19.065337 2716 raft_consensus.cc:2229] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:19.065445 2716 raft_consensus.cc:2258] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:19.065685 2715 tablet_replica.cc:330] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006: stopping tablet replica
I20260620 04:30:19.065734 2717 ts_tablet_manager.cc:1905] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:19.065816 2715 raft_consensus.cc:2229] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:19.066151 2715 raft_consensus.cc:2258] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:19.066848 2716 ts_tablet_manager.cc:1905] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:19.068097 2715 ts_tablet_manager.cc:1905] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:19.068768 2717 ts_tablet_manager.cc:1918] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:30:19.068897 2717 log.cc:1196] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/1f281f365d9a4c0aabfeb2a456d6ed4b
I20260620 04:30:19.069242 2717 ts_tablet_manager.cc:1939] T 1f281f365d9a4c0aabfeb2a456d6ed4b P efa2af3201e74f2583cd893cf5cc869e: Deleting consensus metadata
I20260620 04:30:19.069687 2716 ts_tablet_manager.cc:1918] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:30:19.069769 2716 log.cc:1196] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/1f281f365d9a4c0aabfeb2a456d6ed4b
I20260620 04:30:19.070005 2716 ts_tablet_manager.cc:1939] T 1f281f365d9a4c0aabfeb2a456d6ed4b P fc8b42c0376847348fdbf281a7a986f7: Deleting consensus metadata
I20260620 04:30:19.070183 2114 catalog_manager.cc:4841] TS efa2af3201e74f2583cd893cf5cc869e (127.0.98.3:41729): tablet 1f281f365d9a4c0aabfeb2a456d6ed4b (table test-table [id=71fbe38d855b405b84dd1d4c4bb2bfd5]) successfully deleted
I20260620 04:30:19.070672 2113 catalog_manager.cc:4841] TS fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093): tablet 1f281f365d9a4c0aabfeb2a456d6ed4b (table test-table [id=71fbe38d855b405b84dd1d4c4bb2bfd5]) successfully deleted
I20260620 04:30:19.070698 2715 ts_tablet_manager.cc:1918] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:30:19.070761 2715 log.cc:1196] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/1f281f365d9a4c0aabfeb2a456d6ed4b
I20260620 04:30:19.070991 2715 ts_tablet_manager.cc:1939] T 1f281f365d9a4c0aabfeb2a456d6ed4b P 062b3f3e9c124b8a9e25c830f95de006: Deleting consensus metadata
I20260620 04:30:19.071565 2114 catalog_manager.cc:4841] TS 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331): tablet 1f281f365d9a4c0aabfeb2a456d6ed4b (table test-table [id=71fbe38d855b405b84dd1d4c4bb2bfd5]) successfully deleted
Jun 20 04:30:19 dist-test-slave-rzqm krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929819, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:30:19 dist-test-slave-rzqm krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929819, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:19.091929 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.083443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.0.98.62:44581)
0620 04:30:19.083723 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:19.083732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:19.083818 (+ 86us) server_negotiation.cc:407] Connection header received
0620 04:30:19.083966 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:19.083969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:19.084016 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:19.084086 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:19.084784 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.085404 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.086247 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.086472 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.086557 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.088933 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:19.088936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:19.088939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:19.088964 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:19.090380 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:19.090771 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:19.090774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:19.090776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:19.090853 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:19.091106 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:19.091109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:19.091111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:19.091289 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:19.091414 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:19.091639 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 04:30:19.091762 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20260620 04:30:19.094882 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:19.095340 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:19.100030 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5 (DEFAULT_TABLE table=test-table [id=de5cdf3146c1450ea4f4e843d5f44488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:19.100052 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5 (DEFAULT_TABLE table=test-table [id=de5cdf3146c1450ea4f4e843d5f44488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:19.100131 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5 (DEFAULT_TABLE table=test-table [id=de5cdf3146c1450ea4f4e843d5f44488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:19.100360 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:19.100361 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:19.100379 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:19.102759 2726 tablet_bootstrap.cc:492] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7: Bootstrap starting.
I20260620 04:30:19.103107 2728 tablet_bootstrap.cc:492] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006: Bootstrap starting.
I20260620 04:30:19.103691 2726 tablet_bootstrap.cc:654] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:19.103982 2728 tablet_bootstrap.cc:654] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:19.104557 2727 tablet_bootstrap.cc:492] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e: Bootstrap starting.
I20260620 04:30:19.104921 2726 tablet_bootstrap.cc:492] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7: No bootstrap required, opened a new log
I20260620 04:30:19.105046 2726 ts_tablet_manager.cc:1397] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:19.105398 2727 tablet_bootstrap.cc:654] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:19.105502 2726 raft_consensus.cc:348] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.105623 2726 raft_consensus.cc:374] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:19.105662 2726 raft_consensus.cc:729] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8b42c0376847348fdbf281a7a986f7, State: Initialized, Role: FOLLOWER
I20260620 04:30:19.105832 2726 consensus_queue.cc:260] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.106217 2726 ts_tablet_manager.cc:1428] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:19.106686 2727 tablet_bootstrap.cc:492] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e: No bootstrap required, opened a new log
I20260620 04:30:19.106760 2727 ts_tablet_manager.cc:1397] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:19.106886 2728 tablet_bootstrap.cc:492] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006: No bootstrap required, opened a new log
I20260620 04:30:19.106956 2728 ts_tablet_manager.cc:1397] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 04:30:19.107160 2727 raft_consensus.cc:348] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.107273 2727 raft_consensus.cc:374] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:19.107306 2727 raft_consensus.cc:729] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa2af3201e74f2583cd893cf5cc869e, State: Initialized, Role: FOLLOWER
I20260620 04:30:19.107267 2728 raft_consensus.cc:348] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.107348 2728 raft_consensus.cc:374] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:19.107368 2728 raft_consensus.cc:729] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062b3f3e9c124b8a9e25c830f95de006, State: Initialized, Role: FOLLOWER
I20260620 04:30:19.107430 2727 consensus_queue.cc:260] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.107475 2728 consensus_queue.cc:260] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.107684 2727 ts_tablet_manager.cc:1428] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:30:19.107777 2728 ts_tablet_manager.cc:1428] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:19.147272 2732 raft_consensus.cc:482] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:19.147451 2732 raft_consensus.cc:504] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.148579 2732 leader_election.cc:288] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331), efa2af3201e74f2583cd893cf5cc869e (127.0.98.3:41729)
I20260620 04:30:19.152882 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.148822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40331 (local address 127.0.98.2:55891)
0620 04:30:19.149175 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:19.149190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:19.149296 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:19.149706 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:19.149709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:19.149750 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:19.149878 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:19.149886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.150839 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.150843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:19.151729 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:19.151739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.152439 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.152443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:19.152539 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.152561 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:30:19.152632 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:30:19.152700 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"spinlock_wait_cycles":22016,"thread_start_us":144,"threads_started":1}
I20260620 04:30:19.152889 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.148958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55891 (local address 127.0.98.1:40331)
0620 04:30:19.149281 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:19.149289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:19.149381 (+ 92us) server_negotiation.cc:407] Connection header received
0620 04:30:19.149442 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:19.149445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:19.149493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:19.149579 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:19.150105 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.150726 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.151861 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.152321 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.152401 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.152454 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:19.152657 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 04:30:19.152746 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":135,"threads_started":1}
I20260620 04:30:19.152928 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.149551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48331 (local address 127.0.98.3:41729)
0620 04:30:19.149816 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:19.149824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:19.149838 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:19.149887 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:19.149889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:19.149932 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:19.150005 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:19.150274 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.150923 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.151993 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.152425 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.152526 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.152586 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:19.152735 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 04:30:19.152813 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
I20260620 04:30:19.152916 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.149312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41729 (local address 127.0.98.2:48331)
0620 04:30:19.149643 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:19.149656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:19.149764 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:19.150040 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:19.150042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:19.150058 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:19.150165 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:19.150170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.151042 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.151044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:19.151876 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:19.151886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.152534 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.152536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:19.152609 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.152641 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 04:30:19.152711 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:30:19.152777 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"spinlock_wait_cycles":1408,"thread_start_us":95,"threads_started":1}
I20260620 04:30:19.153558 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a85c3b79dcf4fd9830b75ed3e7b0fd5" candidate_uuid: "fc8b42c0376847348fdbf281a7a986f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062b3f3e9c124b8a9e25c830f95de006" is_pre_election: true
I20260620 04:30:19.153621 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a85c3b79dcf4fd9830b75ed3e7b0fd5" candidate_uuid: "fc8b42c0376847348fdbf281a7a986f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa2af3201e74f2583cd893cf5cc869e" is_pre_election: true
I20260620 04:30:19.153707 2283 raft_consensus.cc:2454] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8b42c0376847348fdbf281a7a986f7 in term 0.
I20260620 04:30:19.153764 2564 raft_consensus.cc:2454] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8b42c0376847348fdbf281a7a986f7 in term 0.
I20260620 04:30:19.154119 2358 leader_election.cc:302] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 062b3f3e9c124b8a9e25c830f95de006, fc8b42c0376847348fdbf281a7a986f7; no voters:
I20260620 04:30:19.154431 2732 raft_consensus.cc:2781] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:19.154488 2732 raft_consensus.cc:482] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:19.154531 2732 raft_consensus.cc:3037] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:19.155572 2732 raft_consensus.cc:504] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.155910 2732 leader_election.cc:288] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [CANDIDATE]: Term 1 election: Requested vote from peers 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331), efa2af3201e74f2583cd893cf5cc869e (127.0.98.3:41729)
I20260620 04:30:19.156455 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a85c3b79dcf4fd9830b75ed3e7b0fd5" candidate_uuid: "fc8b42c0376847348fdbf281a7a986f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa2af3201e74f2583cd893cf5cc869e"
I20260620 04:30:19.156455 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a85c3b79dcf4fd9830b75ed3e7b0fd5" candidate_uuid: "fc8b42c0376847348fdbf281a7a986f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062b3f3e9c124b8a9e25c830f95de006"
I20260620 04:30:19.156598 2283 raft_consensus.cc:3037] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:19.156598 2564 raft_consensus.cc:3037] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:19.157817 2564 raft_consensus.cc:2454] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8b42c0376847348fdbf281a7a986f7 in term 1.
I20260620 04:30:19.157825 2283 raft_consensus.cc:2454] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8b42c0376847348fdbf281a7a986f7 in term 1.
I20260620 04:30:19.158138 2358 leader_election.cc:302] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efa2af3201e74f2583cd893cf5cc869e, fc8b42c0376847348fdbf281a7a986f7; no voters:
I20260620 04:30:19.158322 2732 raft_consensus.cc:2781] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:19.158540 2732 raft_consensus.cc:686] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 1 LEADER]: Becoming Leader. State: Replica: fc8b42c0376847348fdbf281a7a986f7, State: Running, Role: LEADER
I20260620 04:30:19.158780 2732 consensus_queue.cc:237] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.161118 2123 catalog_manager.cc:5495] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fc8b42c0376847348fdbf281a7a986f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } health_report { overall_health: UNKNOWN } } }
I20260620 04:30:19.191668 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.187993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.0.98.2:41093)
0620 04:30:19.188263 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:19.188271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:19.188283 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:30:19.188362 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:19.188365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:19.188417 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:19.188525 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:19.188769 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.189369 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.190209 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.190447 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.190540 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.190804 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:19.191265 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:19.191371 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:19.191453 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 04:30:19.191507 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":97,"threads_started":1}
I20260620 04:30:19.201386 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.198614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35809 (local address 127.0.98.3:41729)
0620 04:30:19.198725 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:19.198728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:19.198742 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:19.198831 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:19.198834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:19.198884 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:19.198946 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:19.199145 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.199647 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.200475 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.200674 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.200740 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.200991 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:19.201089 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:19.201152 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:19.201210 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 04:30:19.201280 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:30:19.201414 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.198408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41729 (local address 127.0.98.2:35809)
0620 04:30:19.198661 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:19.198673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:19.198749 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:19.198958 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:19.198960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:19.198965 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:19.199069 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:19.199074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.199733 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.199735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:19.200388 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:19.200397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.200747 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.200749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:19.200812 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.200855 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:19.201154 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:19.201158 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:30:19.201201 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 04:30:19.201264 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260620 04:30:19.207216 2564 raft_consensus.cc:1261] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Refusing update from remote peer fc8b42c0376847348fdbf281a7a986f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:19.207293 2283 raft_consensus.cc:1261] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Refusing update from remote peer fc8b42c0376847348fdbf281a7a986f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:19.207726 2732 consensus_queue.cc:1035] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:19.207934 2737 consensus_queue.cc:1035] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:19.225340 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.222099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41093 (local address 127.0.98.3:38737)
0620 04:30:19.222379 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:19.222394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:19.222500 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:19.222727 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:19.222730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:19.222737 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:19.222841 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:19.222847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.223544 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.223548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:19.224175 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:19.224184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.224612 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.224616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:19.224698 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.224727 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:19.225041 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:19.225047 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:30:19.225109 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 04:30:19.225194 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":109,"threads_started":1}
I20260620 04:30:19.225322 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.222140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38737 (local address 127.0.98.2:41093)
0620 04:30:19.222269 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:19.222274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:19.222468 (+ 194us) server_negotiation.cc:407] Connection header received
0620 04:30:19.222582 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:19.222585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:19.222643 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:19.222708 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:19.222932 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.223438 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.224285 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.224512 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.224612 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.224846 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:19.224937 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:19.225043 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:19.225123 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 04:30:19.225183 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:30:19.248042 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:19.248219 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:19.250494 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78: Sending DeleteTablet for 3 replicas of tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5
I20260620 04:30:19.251075 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45586
I20260620 04:30:19.251222 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36092
I20260620 04:30:19.251271 2715 tablet_replica.cc:330] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006: stopping tablet replica
I20260620 04:30:19.251400 2716 tablet_replica.cc:330] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7: stopping tablet replica
I20260620 04:30:19.251394 2715 raft_consensus.cc:2229] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:19.251509 2715 raft_consensus.cc:2258] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:19.251502 2716 raft_consensus.cc:2229] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:19.251814 2716 raft_consensus.cc:2258] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:19.251837 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:19.252247 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:19.252473 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56912
I20260620 04:30:19.252601 2716 ts_tablet_manager.cc:1905] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:19.252648 2717 tablet_replica.cc:330] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e: stopping tablet replica
I20260620 04:30:19.252755 2717 raft_consensus.cc:2229] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:19.252866 2717 raft_consensus.cc:2258] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:19.253750 2715 ts_tablet_manager.cc:1905] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:19.253978 2717 ts_tablet_manager.cc:1905] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:19.255120 2716 ts_tablet_manager.cc:1918] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:19.255192 2716 log.cc:1196] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/7a85c3b79dcf4fd9830b75ed3e7b0fd5
I20260620 04:30:19.255430 2716 ts_tablet_manager.cc:1939] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P fc8b42c0376847348fdbf281a7a986f7: Deleting consensus metadata
I20260620 04:30:19.256314 2715 ts_tablet_manager.cc:1918] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:19.256394 2715 log.cc:1196] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/7a85c3b79dcf4fd9830b75ed3e7b0fd5
I20260620 04:30:19.256422 2717 ts_tablet_manager.cc:1918] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:19.256489 2717 log.cc:1196] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/7a85c3b79dcf4fd9830b75ed3e7b0fd5
I20260620 04:30:19.256624 2715 ts_tablet_manager.cc:1939] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P 062b3f3e9c124b8a9e25c830f95de006: Deleting consensus metadata
I20260620 04:30:19.256703 2717 ts_tablet_manager.cc:1939] T 7a85c3b79dcf4fd9830b75ed3e7b0fd5 P efa2af3201e74f2583cd893cf5cc869e: Deleting consensus metadata
I20260620 04:30:19.257211 2113 catalog_manager.cc:4841] TS fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093): tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5 (table test-table [id=de5cdf3146c1450ea4f4e843d5f44488]) successfully deleted
I20260620 04:30:19.257472 2114 catalog_manager.cc:4841] TS efa2af3201e74f2583cd893cf5cc869e (127.0.98.3:41729): tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5 (table test-table [id=de5cdf3146c1450ea4f4e843d5f44488]) successfully deleted
I20260620 04:30:19.257606 2114 catalog_manager.cc:4841] TS 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331): tablet 7a85c3b79dcf4fd9830b75ed3e7b0fd5 (table test-table [id=de5cdf3146c1450ea4f4e843d5f44488]) successfully deleted
I20260620 04:30:19.260037 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 33b97c2970004899a57bac7658ca8898 (DEFAULT_TABLE table=test-table [id=325be5874af54ef395a480a8a1112937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:19.260105 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 33b97c2970004899a57bac7658ca8898 (DEFAULT_TABLE table=test-table [id=325be5874af54ef395a480a8a1112937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:19.260294 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b97c2970004899a57bac7658ca8898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:19.260351 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b97c2970004899a57bac7658ca8898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:19.262414 2726 tablet_bootstrap.cc:492] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7: Bootstrap starting.
I20260620 04:30:19.262506 2728 tablet_bootstrap.cc:492] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006: Bootstrap starting.
I20260620 04:30:19.262916 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 33b97c2970004899a57bac7658ca8898 (DEFAULT_TABLE table=test-table [id=325be5874af54ef395a480a8a1112937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:19.263161 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b97c2970004899a57bac7658ca8898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:19.263329 2726 tablet_bootstrap.cc:654] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:19.263334 2728 tablet_bootstrap.cc:654] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:19.265381 2727 tablet_bootstrap.cc:492] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e: Bootstrap starting.
I20260620 04:30:19.266144 2727 tablet_bootstrap.cc:654] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:19.266201 2726 tablet_bootstrap.cc:492] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7: No bootstrap required, opened a new log
I20260620 04:30:19.266273 2726 ts_tablet_manager.cc:1397] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 04:30:19.266368 2728 tablet_bootstrap.cc:492] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006: No bootstrap required, opened a new log
I20260620 04:30:19.266445 2728 ts_tablet_manager.cc:1397] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 04:30:19.266714 2726 raft_consensus.cc:348] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.266840 2726 raft_consensus.cc:374] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:19.266875 2726 raft_consensus.cc:729] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8b42c0376847348fdbf281a7a986f7, State: Initialized, Role: FOLLOWER
I20260620 04:30:19.266811 2728 raft_consensus.cc:348] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.266932 2728 raft_consensus.cc:374] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:19.266961 2728 raft_consensus.cc:729] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062b3f3e9c124b8a9e25c830f95de006, State: Initialized, Role: FOLLOWER
I20260620 04:30:19.266990 2726 consensus_queue.cc:260] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.267071 2728 consensus_queue.cc:260] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.267264 2726 ts_tablet_manager.cc:1428] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:19.267305 2728 ts_tablet_manager.cc:1428] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:19.268090 2727 tablet_bootstrap.cc:492] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e: No bootstrap required, opened a new log
I20260620 04:30:19.268177 2727 ts_tablet_manager.cc:1397] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:19.268569 2727 raft_consensus.cc:348] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.268676 2727 raft_consensus.cc:374] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:19.268707 2727 raft_consensus.cc:729] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa2af3201e74f2583cd893cf5cc869e, State: Initialized, Role: FOLLOWER
I20260620 04:30:19.268821 2727 consensus_queue.cc:260] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.269109 2727 ts_tablet_manager.cc:1428] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:19.509222 2692 raft_consensus.cc:482] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:19.509382 2692 raft_consensus.cc:504] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.509868 2692 leader_election.cc:288] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331), fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093)
I20260620 04:30:19.510375 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b97c2970004899a57bac7658ca8898" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8b42c0376847348fdbf281a7a986f7" is_pre_election: true
I20260620 04:30:19.510377 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b97c2970004899a57bac7658ca8898" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062b3f3e9c124b8a9e25c830f95de006" is_pre_election: true
I20260620 04:30:19.510529 2424 raft_consensus.cc:2454] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 0.
I20260620 04:30:19.510532 2283 raft_consensus.cc:2454] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 0.
I20260620 04:30:19.510831 2498 leader_election.cc:302] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 062b3f3e9c124b8a9e25c830f95de006, efa2af3201e74f2583cd893cf5cc869e; no voters:
I20260620 04:30:19.510995 2692 raft_consensus.cc:2781] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:19.511056 2692 raft_consensus.cc:482] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:19.511082 2692 raft_consensus.cc:3037] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:19.512025 2692 raft_consensus.cc:504] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.512337 2692 leader_election.cc:288] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 election: Requested vote from peers 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331), fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093)
I20260620 04:30:19.512710 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b97c2970004899a57bac7658ca8898" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "062b3f3e9c124b8a9e25c830f95de006"
I20260620 04:30:19.512733 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b97c2970004899a57bac7658ca8898" candidate_uuid: "efa2af3201e74f2583cd893cf5cc869e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8b42c0376847348fdbf281a7a986f7"
I20260620 04:30:19.512820 2283 raft_consensus.cc:3037] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:19.512852 2424 raft_consensus.cc:3037] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:19.513751 2283 raft_consensus.cc:2454] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 1.
I20260620 04:30:19.513840 2424 raft_consensus.cc:2454] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa2af3201e74f2583cd893cf5cc869e in term 1.
I20260620 04:30:19.514019 2498 leader_election.cc:302] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 062b3f3e9c124b8a9e25c830f95de006, efa2af3201e74f2583cd893cf5cc869e; no voters:
I20260620 04:30:19.514161 2692 raft_consensus.cc:2781] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:19.514248 2692 raft_consensus.cc:686] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 1 LEADER]: Becoming Leader. State: Replica: efa2af3201e74f2583cd893cf5cc869e, State: Running, Role: LEADER
I20260620 04:30:19.514382 2692 consensus_queue.cc:237] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } }
I20260620 04:30:19.516058 2123 catalog_manager.cc:5495] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e reported cstate change: term changed from 0 to 1, leader changed from <none> to efa2af3201e74f2583cd893cf5cc869e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "efa2af3201e74f2583cd893cf5cc869e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa2af3201e74f2583cd893cf5cc869e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41729 } health_report { overall_health: HEALTHY } } }
I20260620 04:30:19.566797 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:19.563715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.0.98.3:41729)
0620 04:30:19.563823 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:19.563826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:19.563851 (+ 25us) server_negotiation.cc:407] Connection header received
0620 04:30:19.563963 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:19.563965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:19.564004 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:19.564066 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:19.564368 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.564991 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.565940 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:19.566154 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:19.566235 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:19.566452 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:19.566523 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:19.566569 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:19.566638 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 04:30:19.566681 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:30:19.572708 2424 raft_consensus.cc:1261] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Refusing update from remote peer efa2af3201e74f2583cd893cf5cc869e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:19.572804 2283 raft_consensus.cc:1261] T 33b97c2970004899a57bac7658ca8898 P 062b3f3e9c124b8a9e25c830f95de006 [term 1 FOLLOWER]: Refusing update from remote peer efa2af3201e74f2583cd893cf5cc869e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:19.573153 2692 consensus_queue.cc:1035] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8b42c0376847348fdbf281a7a986f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:19.573312 2696 consensus_queue.cc:1035] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [LEADER]: Connected to new peer: Peer: permanent_uuid: "062b3f3e9c124b8a9e25c830f95de006" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 04:30:19.586879 2694 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 04:30:21.137085 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:21.137257 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:21.140017 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f5a905b3dcd48559d1a3ea6d3877b78: Sending DeleteTablet for 3 replicas of tablet 33b97c2970004899a57bac7658ca8898
I20260620 04:30:21.140719 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b97c2970004899a57bac7658ca8898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45586
I20260620 04:30:21.140719 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b97c2970004899a57bac7658ca8898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56912
I20260620 04:30:21.141083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 2199
I20260620 04:30:21.141089 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b97c2970004899a57bac7658ca8898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36092
I20260620 04:30:21.142462 2779 tablet_replica.cc:330] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7: stopping tablet replica
I20260620 04:30:21.142613 2779 raft_consensus.cc:2229] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:21.142750 2779 raft_consensus.cc:2258] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:21.143397 2779 ts_tablet_manager.cc:1905] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:21.145632 2779 ts_tablet_manager.cc:1918] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:30:21.145699 2779 log.cc:1196] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/33b97c2970004899a57bac7658ca8898
I20260620 04:30:21.145902 2779 ts_tablet_manager.cc:1939] T 33b97c2970004899a57bac7658ca8898 P fc8b42c0376847348fdbf281a7a986f7: Deleting consensus metadata
I20260620 04:30:21.146596 2113 catalog_manager.cc:4841] TS fc8b42c0376847348fdbf281a7a986f7 (127.0.98.2:41093): tablet 33b97c2970004899a57bac7658ca8898 (table test-table [id=325be5874af54ef395a480a8a1112937]) successfully deleted
I20260620 04:30:21.147241 2778 tablet_replica.cc:330] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e: stopping tablet replica
I20260620 04:30:21.147358 2778 raft_consensus.cc:2229] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:21.147528 2778 raft_consensus.cc:2258] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:21.148180 2778 ts_tablet_manager.cc:1905] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:30:21.149305 2114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40331 (error 108)
I20260620 04:30:21.149521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 2339
I20260620 04:30:21.151003 2778 ts_tablet_manager.cc:1918] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 04:30:21.151070 2778 log.cc:1196] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/33b97c2970004899a57bac7658ca8898
I20260620 04:30:21.151304 2778 ts_tablet_manager.cc:1939] T 33b97c2970004899a57bac7658ca8898 P efa2af3201e74f2583cd893cf5cc869e: Deleting consensus metadata
I20260620 04:30:21.152140 2114 catalog_manager.cc:4841] TS efa2af3201e74f2583cd893cf5cc869e (127.0.98.3:41729): tablet 33b97c2970004899a57bac7658ca8898 (table test-table [id=325be5874af54ef395a480a8a1112937]) successfully deleted
I20260620 04:30:21.152262 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:21.151841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40331 (local address 127.0.0.1:47686)
0620 04:30:21.152083 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:21.152169 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
W20260620 04:30:21.152429 2114 catalog_manager.cc:4584] TS 062b3f3e9c124b8a9e25c830f95de006 (127.0.98.1:40331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33b97c2970004899a57bac7658ca8898: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40331: connect: Connection refused (error 111)
I20260620 04:30:21.156275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 2479
I20260620 04:30:21.162127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 2095
2026-06-20T04:30:21Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6407 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:21 dist-test-slave-rzqm krb5kdc[2784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:21 dist-test-slave-rzqm krb5kdc[2784](info): set up 2 sockets
Jun 20 04:30:21 dist-test-slave-rzqm krb5kdc[2784](info): commencing operation
krb5kdc: starting...
W20260620 04:30:22.116807 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:22 dist-test-slave-rzqm krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:22.232941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:22.342195 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:22.342487 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:22.342545 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:22.346652 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:22.346724 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:22.346750 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:22.346771 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:22.346797 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:22.352298 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:22.353473 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:22.354511 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:22.363032 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:22.363128 2797 server_base.cc:1034] running on GCE node
W20260620 04:30:22.363037 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:22.363055 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:22.363842 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:22.364995 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:22.366189 2797 hybrid_clock.cc:648] HybridClock initialized: now 1781929822366162 us; error 40 us; skew 500 ppm
Jun 20 04:30:22 dist-test-slave-rzqm krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:22.378491 2797 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:22.379599 2797 webserver.cc:460] Webserver started at http://127.0.98.62:33735/ using document root <none> and password file <none>
I20260620 04:30:22.380185 2797 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:22.380250 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:22.380467 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:22.382664 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "427814dfb6274dd59eb50095c9eadc00"
format_stamp: "Formatted at 2026-06-20 04:30:22 on dist-test-slave-rzqm"
server_key: "b7eed95db262e6af7ab0e170eaf3be7c"
server_key_iv: "7f189f212bee5cf02fbd4a61800e7ff5"
server_key_version: "encryptionkey@0"
I20260620 04:30:22.383232 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "427814dfb6274dd59eb50095c9eadc00"
format_stamp: "Formatted at 2026-06-20 04:30:22 on dist-test-slave-rzqm"
server_key: "b7eed95db262e6af7ab0e170eaf3be7c"
server_key_iv: "7f189f212bee5cf02fbd4a61800e7ff5"
server_key_version: "encryptionkey@0"
I20260620 04:30:22.387163 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 04:30:22.389655 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:22.390977 2797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:22.391139 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "427814dfb6274dd59eb50095c9eadc00"
format_stamp: "Formatted at 2026-06-20 04:30:22 on dist-test-slave-rzqm"
server_key: "b7eed95db262e6af7ab0e170eaf3be7c"
server_key_iv: "7f189f212bee5cf02fbd4a61800e7ff5"
server_key_version: "encryptionkey@0"
I20260620 04:30:22.391263 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:22.415009 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:22.418341 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:22.418540 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:22.425587 2797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45091
I20260620 04:30:22.425608 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45091 every 8 connection(s)
I20260620 04:30:22.426592 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:22.429381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 2797
I20260620 04:30:22.429399 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:22.429468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:22.429742 392 external_mini_cluster.cc:1428] Setting key 9dc4f3779848cc85509acb5ac0d99456
I20260620 04:30:22.436088 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00: Bootstrap starting.
I20260620 04:30:22.438808 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00: Neither blocks nor log segments found. Creating new log.
Jun 20 04:30:22 dist-test-slave-rzqm krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:22.439590 2865 log.cc:824] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:22.441721 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00: No bootstrap required, opened a new log
I20260620 04:30:22.444429 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:22.431053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51788 (local address 127.0.98.62:45091)
0620 04:30:22.431439 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:22.431454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:22.431489 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:30:22.432420 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:22.432442 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:22.432820 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:22.433306 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:22.434337 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:22.435382 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:22.436518 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:22.437060 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:22.437251 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:22.439931 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:22.439938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:22.439959 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:22.439993 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:22.441667 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:22.442110 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:22.442116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:22.442121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:22.442239 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:22.442500 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:22.442504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:22.442505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:22.442904 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:22.443082 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:22.443370 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 04:30:22.443655 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260620 04:30:22.445420 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427814dfb6274dd59eb50095c9eadc00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260620 04:30:22.445719 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:22.445782 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427814dfb6274dd59eb50095c9eadc00, State: Initialized, Role: FOLLOWER
I20260620 04:30:22.446251 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427814dfb6274dd59eb50095c9eadc00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260620 04:30:22.446412 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:22.446472 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:22.446573 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:22.447479 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427814dfb6274dd59eb50095c9eadc00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260620 04:30:22.447803 2865 leader_election.cc:302] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 427814dfb6274dd59eb50095c9eadc00; no voters:
I20260620 04:30:22.448122 2865 leader_election.cc:288] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:22.448238 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:22.448459 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [term 1 LEADER]: Becoming Leader. State: Replica: 427814dfb6274dd59eb50095c9eadc00, State: Running, Role: LEADER
I20260620 04:30:22.448815 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427814dfb6274dd59eb50095c9eadc00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } }
I20260620 04:30:22.449342 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:22.450445 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 427814dfb6274dd59eb50095c9eadc00. Latest consensus state: current_term: 1 leader_uuid: "427814dfb6274dd59eb50095c9eadc00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427814dfb6274dd59eb50095c9eadc00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } } }
I20260620 04:30:22.450589 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:22.450851 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "427814dfb6274dd59eb50095c9eadc00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427814dfb6274dd59eb50095c9eadc00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45091 } } }
I20260620 04:30:22.450977 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:22.451017 2879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:22.454103 2879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:22.459777 2879 catalog_manager.cc:1347] Generated new cluster ID: bddb93e066dd455aa2b9e48d316d1410
I20260620 04:30:22.459863 2879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:22.476279 2879 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:22.477347 2879 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:22.491863 2879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 427814dfb6274dd59eb50095c9eadc00: Generated new TSK 0
I20260620 04:30:22.492733 2879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:30:22.530191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:22.642088 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:22.642396 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:22.642457 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:22.646765 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:22.646842 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:22.646927 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:22.652580 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:22.653735 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:22.654819 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:22.664482 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:22.664508 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:22.664724 2891 server_base.cc:1034] running on GCE node
W20260620 04:30:22.664484 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:22.665298 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:22.665977 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:22.667146 2891 hybrid_clock.cc:648] HybridClock initialized: now 1781929822667124 us; error 33 us; skew 500 ppm
Jun 20 04:30:22 dist-test-slave-rzqm krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:22.669942 2891 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:30:22.670964 2891 webserver.cc:460] Webserver started at http://127.0.98.1:46563/ using document root <none> and password file <none>
I20260620 04:30:22.671473 2891 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:22.671530 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:22.671713 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:22.673779 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "d0182993539d43159706b0c29028f51b"
format_stamp: "Formatted at 2026-06-20 04:30:22 on dist-test-slave-rzqm"
server_key: "8fd28e88a860c5179300d8b144d868e2"
server_key_iv: "5add804646fedcc35b8f1b90fe532017"
server_key_version: "encryptionkey@0"
I20260620 04:30:22.674263 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d0182993539d43159706b0c29028f51b"
format_stamp: "Formatted at 2026-06-20 04:30:22 on dist-test-slave-rzqm"
server_key: "8fd28e88a860c5179300d8b144d868e2"
server_key_iv: "5add804646fedcc35b8f1b90fe532017"
server_key_version: "encryptionkey@0"
I20260620 04:30:22.677984 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:22.680382 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:22.681779 2891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:22.681903 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "d0182993539d43159706b0c29028f51b"
format_stamp: "Formatted at 2026-06-20 04:30:22 on dist-test-slave-rzqm"
server_key: "8fd28e88a860c5179300d8b144d868e2"
server_key_iv: "5add804646fedcc35b8f1b90fe532017"
server_key_version: "encryptionkey@0"
I20260620 04:30:22.682003 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:22.701567 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:22.704562 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:22.704741 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:22.705361 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:22.706189 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:22.706238 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:22.706288 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:22.706314 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:22.714932 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34589
I20260620 04:30:22.715004 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34589 every 8 connection(s)
I20260620 04:30:22.715834 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:22.716847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 2891
I20260620 04:30:22.716936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:22.717216 392 external_mini_cluster.cc:1428] Setting key a5f8a4a2824aef3db92af29b6ef242c8
Jun 20 04:30:22 dist-test-slave-rzqm krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:30:22.729837 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:22.718707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41025 (local address 127.0.98.62:45091)
0620 04:30:22.718825 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:22.718829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:22.718843 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:22.720151 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:22.720155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:22.720207 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:22.720279 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:22.721525 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:22.722284 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:22.723536 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:22.723895 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:22.724024 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:22.726413 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:22.726417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:22.726420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:22.726447 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:22.727670 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:22.728272 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:22.728275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:22.728277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:22.728359 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:22.728732 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:22.728735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:22.728737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:22.728925 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:22.729033 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:22.729539 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 04:30:22.729674 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:30:22.730732 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:22.718151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45091 (local address 127.0.98.1:41025)
0620 04:30:22.718658 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:22.718718 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:22.719947 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:22.720468 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:22.720475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:22.720828 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:22.721380 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:22.721394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:22.722421 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:22.722424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:22.723406 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:22.723416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:22.724042 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:22.724045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:22.724189 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:22.724777 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:22.724793 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:22.726248 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:22.727794 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:22.727799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:22.727811 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:22.728165 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:22.728438 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:22.728440 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:22.728441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:22.728622 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:22.729043 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:22.729050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:22.729309 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 04:30:22.729481 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 04:30:22.729806 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":114,"threads_started":1}
I20260620 04:30:22.731976 3020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45091
I20260620 04:30:22.732210 3020 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:22.732674 3020 heartbeater.cc:511] Master 127.0.98.62:45091 requested a full tablet report, sending...
I20260620 04:30:22.734377 2829 ts_manager.cc:194] Registered new tserver with Master: d0182993539d43159706b0c29028f51b (127.0.98.1:34589)
I20260620 04:30:22.735526 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:30:22.742305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:22.853024 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:22.853330 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:22.853394 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:22.857506 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:22.857578 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:22.857662 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:22.863540 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:22.864696 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:22.865846 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:30:22.876021 3025 server_base.cc:1034] running on GCE node
W20260620 04:30:22.875947 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:22.875941 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:22.876184 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:22.876670 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:22.877399 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:22.878575 3025 hybrid_clock.cc:648] HybridClock initialized: now 1781929822878538 us; error 48 us; skew 500 ppm
Jun 20 04:30:22 dist-test-slave-rzqm krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:22.881425 3025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:30:22.882350 3025 webserver.cc:460] Webserver started at http://127.0.98.2:38923/ using document root <none> and password file <none>
I20260620 04:30:22.882898 3025 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:22.882968 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:22.883194 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:22.885387 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "49234e4339a341d88186c472b7cff2e3"
format_stamp: "Formatted at 2026-06-20 04:30:22 on dist-test-slave-rzqm"
server_key: "73768169d2b9214f620d62cdf1df8227"
server_key_iv: "d2243910102b2db763cb64bbeddcd8ab"
server_key_version: "encryptionkey@0"
I20260620 04:30:22.885949 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "49234e4339a341d88186c472b7cff2e3"
format_stamp: "Formatted at 2026-06-20 04:30:22 on dist-test-slave-rzqm"
server_key: "73768169d2b9214f620d62cdf1df8227"
server_key_iv: "d2243910102b2db763cb64bbeddcd8ab"
server_key_version: "encryptionkey@0"
I20260620 04:30:22.889662 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:22.892175 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:22.893510 3025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:22.893625 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "49234e4339a341d88186c472b7cff2e3"
format_stamp: "Formatted at 2026-06-20 04:30:22 on dist-test-slave-rzqm"
server_key: "73768169d2b9214f620d62cdf1df8227"
server_key_iv: "d2243910102b2db763cb64bbeddcd8ab"
server_key_version: "encryptionkey@0"
I20260620 04:30:22.893724 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:22.932677 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:22.936305 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:22.936510 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:22.937166 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:22.938023 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:22.938071 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:22.938123 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:22.938158 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:22.946951 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41065
I20260620 04:30:22.946976 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41065 every 8 connection(s)
I20260620 04:30:22.947885 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:22.948823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 3025
I20260620 04:30:22.948922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:22.949215 392 external_mini_cluster.cc:1428] Setting key 595cab43f8930b65482748e7dbf5a80d
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 20 04:30:22 dist-test-slave-rzqm krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:22.966337 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:22.949981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55885 (local address 127.0.98.62:45091)
0620 04:30:22.950165 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:22.950170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:22.951400 (+ 1230us) server_negotiation.cc:407] Connection header received
0620 04:30:22.953110 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:22.953114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:22.953192 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:22.953282 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:22.954539 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:22.955423 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:22.957389 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:22.957624 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:22.957719 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:22.962869 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:22.962872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:22.962874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:22.962902 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:22.964239 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:22.964764 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:22.964767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:22.964769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:22.964846 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:22.965212 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:22.965215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:22.965216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:22.965403 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:22.965487 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:22.966028 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 04:30:22.966169 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 04:30:22.967252 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:22.950554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45091 (local address 127.0.98.2:55885)
0620 04:30:22.951197 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:22.951297 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:22.952853 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:22.953440 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:22.953447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:22.953836 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:22.954358 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:22.954374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:22.955584 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:22.955588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:22.957241 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:22.957254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:22.957805 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:22.957810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:22.958032 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:22.958701 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:22.958726 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:22.962666 (+ 3940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:22.964360 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:22.964367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:22.964387 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:22.964652 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:22.964925 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:22.964929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:22.964931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:22.965117 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:22.965499 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:22.965507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:22.965785 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 04:30:22.965955 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 04:30:22.966335 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260620 04:30:22.968578 3154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45091
I20260620 04:30:22.968845 3154 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:22.969383 3154 heartbeater.cc:511] Master 127.0.98.62:45091 requested a full tablet report, sending...
I20260620 04:30:22.970475 2829 ts_manager.cc:194] Registered new tserver with Master: 49234e4339a341d88186c472b7cff2e3 (127.0.98.2:41065)
I20260620 04:30:22.971042 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:30:22.976645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:23.087919 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:23.088192 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:23.088244 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:23.092374 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:23.092447 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:23.092507 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:23.097923 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:23.099064 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:23.100042 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:30:23.109362 3159 server_base.cc:1034] running on GCE node
W20260620 04:30:23.109324 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:23.109345 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:23.109324 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:23.110146 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:23.110797 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:23.111984 3159 hybrid_clock.cc:648] HybridClock initialized: now 1781929823111953 us; error 46 us; skew 500 ppm
Jun 20 04:30:23 dist-test-slave-rzqm krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:23.114823 3159 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:30:23.115748 3159 webserver.cc:460] Webserver started at http://127.0.98.3:46565/ using document root <none> and password file <none>
I20260620 04:30:23.116290 3159 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:23.116357 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:23.116541 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:23.118657 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "c9fbc2f551a8460f81d0630e90b33fef"
format_stamp: "Formatted at 2026-06-20 04:30:23 on dist-test-slave-rzqm"
server_key: "b496c4641d53ff1ed784772f399b8b0d"
server_key_iv: "bb1289c714efc726f2952ab051ca0fe5"
server_key_version: "encryptionkey@0"
I20260620 04:30:23.119204 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c9fbc2f551a8460f81d0630e90b33fef"
format_stamp: "Formatted at 2026-06-20 04:30:23 on dist-test-slave-rzqm"
server_key: "b496c4641d53ff1ed784772f399b8b0d"
server_key_iv: "bb1289c714efc726f2952ab051ca0fe5"
server_key_version: "encryptionkey@0"
I20260620 04:30:23.122954 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:30:23.125622 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:23.127048 3159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:23.127184 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "c9fbc2f551a8460f81d0630e90b33fef"
format_stamp: "Formatted at 2026-06-20 04:30:23 on dist-test-slave-rzqm"
server_key: "b496c4641d53ff1ed784772f399b8b0d"
server_key_iv: "bb1289c714efc726f2952ab051ca0fe5"
server_key_version: "encryptionkey@0"
I20260620 04:30:23.127295 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:23.149811 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:23.153201 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:23.153391 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:23.153985 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:23.154826 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:23.154876 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:23.154930 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:23.154958 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:23.163343 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44637
I20260620 04:30:23.163365 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44637 every 8 connection(s)
I20260620 04:30:23.164326 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:23.173353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 3159
I20260620 04:30:23.173471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:23.173775 392 external_mini_cluster.cc:1428] Setting key 9ebcee4e3779d534fdae5d0513b1a127
Jun 20 04:30:23 dist-test-slave-rzqm krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:23.178686 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:23.166135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33909 (local address 127.0.98.62:45091)
0620 04:30:23.166281 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:23.166286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:23.167239 (+ 953us) server_negotiation.cc:407] Connection header received
0620 04:30:23.168629 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:23.168633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:23.168683 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:23.168762 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:23.170045 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:23.170580 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:23.171905 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:23.172126 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:23.172231 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:23.174980 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:23.174985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:23.174987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:23.175025 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:23.176548 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:23.177082 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:23.177126 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:23.177128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:23.177213 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:23.177582 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:23.177585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:23.177586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:23.177786 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:23.177867 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:23.178447 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 04:30:23.178565 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:30:23.179512 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:23.166618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45091 (local address 127.0.98.3:33909)
0620 04:30:23.167081 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:23.167127 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:23.168413 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:23.168934 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:23.168941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:23.169392 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:23.169901 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:23.169915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:23.170708 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:23.170712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:23.171746 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:23.171756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:23.172295 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:23.172299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:23.172455 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:23.173030 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:23.173055 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:23.174811 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:23.176659 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:23.176664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:23.176678 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:23.176959 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:23.177310 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:23.177312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:23.177314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:23.177457 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:23.177890 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:23.177896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:23.178154 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 04:30:23.178313 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 04:30:23.178583 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
I20260620 04:30:23.180660 3288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45091
I20260620 04:30:23.180881 3288 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:23.181398 3288 heartbeater.cc:511] Master 127.0.98.62:45091 requested a full tablet report, sending...
I20260620 04:30:23.182361 2829 ts_manager.cc:194] Registered new tserver with Master: c9fbc2f551a8460f81d0630e90b33fef (127.0.98.3:44637)
I20260620 04:30:23.182910 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33909
I20260620 04:30:23.187620 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 04:30:23.200369 2868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:23.198842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.0.98.62:45091)
0620 04:30:23.198990 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:23.198994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:23.199229 (+ 235us) server_negotiation.cc:407] Connection header received
0620 04:30:23.199284 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:23.199287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:23.199340 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:23.199423 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:23.200232 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51794: BlockingRecv error: recv got EOF from 127.0.0.1:51794 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 04:30:23.201386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 2891
I20260620 04:30:23.206637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 3025
I20260620 04:30:23.211454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 3159
I20260620 04:30:23.215932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 2797
2026-06-20T04:30:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2052 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:23 dist-test-slave-rzqm krb5kdc[3300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:23 dist-test-slave-rzqm krb5kdc[3300](info): set up 2 sockets
Jun 20 04:30:23 dist-test-slave-rzqm krb5kdc[3300](info): commencing operation
krb5kdc: starting...
W20260620 04:30:24.164863 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:24 dist-test-slave-rzqm krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:24.284305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38265
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:24.395361 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:24.395664 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:24.395706 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:24.399861 3313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:24.399940 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:24.399959 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:24.399976 3313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:24.399991 3313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:24.405635 3313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:24.406783 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:24.407823 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:24.416431 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:24.416471 3313 server_base.cc:1034] running on GCE node
W20260620 04:30:24.416430 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:24.416445 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:24.417284 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:24.418444 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:24.419638 3313 hybrid_clock.cc:648] HybridClock initialized: now 1781929824419623 us; error 31 us; skew 500 ppm
Jun 20 04:30:24 dist-test-slave-rzqm krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:24.422483 3313 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:24.423429 3313 webserver.cc:460] Webserver started at http://127.0.98.62:35647/ using document root <none> and password file <none>
I20260620 04:30:24.423983 3313 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:24.424050 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:24.424260 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:24.426401 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "a2a513483baa4a7da25eb7ed8daef5c9"
format_stamp: "Formatted at 2026-06-20 04:30:24 on dist-test-slave-rzqm"
server_key: "283d5221da77e57e13db9c719961b88f"
server_key_iv: "9c19750dac89440c20e3d6949c1ca9b0"
server_key_version: "encryptionkey@0"
I20260620 04:30:24.426944 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2a513483baa4a7da25eb7ed8daef5c9"
format_stamp: "Formatted at 2026-06-20 04:30:24 on dist-test-slave-rzqm"
server_key: "283d5221da77e57e13db9c719961b88f"
server_key_iv: "9c19750dac89440c20e3d6949c1ca9b0"
server_key_version: "encryptionkey@0"
I20260620 04:30:24.430759 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:30:24.433523 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:24.434890 3313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:24.435017 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "a2a513483baa4a7da25eb7ed8daef5c9"
format_stamp: "Formatted at 2026-06-20 04:30:24 on dist-test-slave-rzqm"
server_key: "283d5221da77e57e13db9c719961b88f"
server_key_iv: "9c19750dac89440c20e3d6949c1ca9b0"
server_key_version: "encryptionkey@0"
I20260620 04:30:24.435129 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:24.454779 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:24.458532 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:24.458722 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:24.465737 3313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38265
I20260620 04:30:24.465998 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38265 every 8 connection(s)
I20260620 04:30:24.466722 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:24.469599 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:24.470659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 3313
I20260620 04:30:24.470773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:24.471033 392 external_mini_cluster.cc:1428] Setting key 0217780bf05dcf5439f1b65bb34b92a5
I20260620 04:30:24.476675 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9: Bootstrap starting.
I20260620 04:30:24.479590 3381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9: Neither blocks nor log segments found. Creating new log.
Jun 20 04:30:24 dist-test-slave-rzqm krb5kdc[3300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:24.480415 3381 log.cc:824] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:24.483139 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9: No bootstrap required, opened a new log
I20260620 04:30:24.486366 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:24.472219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.0.98.62:38265)
0620 04:30:24.472647 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:24.472663 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:24.472702 (+ 39us) server_negotiation.cc:407] Connection header received
0620 04:30:24.473472 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:24.473494 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:24.473868 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:24.474196 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:24.474998 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:24.476006 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:24.476880 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:24.477436 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:24.477559 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:24.480712 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:24.480720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:24.480741 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:24.480780 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:24.482877 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:24.483401 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:24.483407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:24.483414 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:24.483573 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:24.483871 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:24.483874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:24.483875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:24.484433 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:24.484620 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:24.484924 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 04:30:24.485312 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260620 04:30:24.487706 3381 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a513483baa4a7da25eb7ed8daef5c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38265 } }
I20260620 04:30:24.487983 3381 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:24.488045 3381 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a513483baa4a7da25eb7ed8daef5c9, State: Initialized, Role: FOLLOWER
I20260620 04:30:24.488490 3381 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a513483baa4a7da25eb7ed8daef5c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38265 } }
I20260620 04:30:24.488648 3381 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:24.488718 3381 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:24.488811 3381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:24.489431 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:30:24.489991 3381 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a513483baa4a7da25eb7ed8daef5c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38265 } }
I20260620 04:30:24.490351 3381 leader_election.cc:302] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a513483baa4a7da25eb7ed8daef5c9; no voters:
I20260620 04:30:24.490737 3381 leader_election.cc:288] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:24.490867 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:24.491080 3386 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [term 1 LEADER]: Becoming Leader. State: Replica: a2a513483baa4a7da25eb7ed8daef5c9, State: Running, Role: LEADER
I20260620 04:30:24.491524 3386 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a513483baa4a7da25eb7ed8daef5c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38265 } }
I20260620 04:30:24.492019 3381 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:24.493337 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a513483baa4a7da25eb7ed8daef5c9. Latest consensus state: current_term: 1 leader_uuid: "a2a513483baa4a7da25eb7ed8daef5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a513483baa4a7da25eb7ed8daef5c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38265 } } }
I20260620 04:30:24.493299 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a513483baa4a7da25eb7ed8daef5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a513483baa4a7da25eb7ed8daef5c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38265 } } }
I20260620 04:30:24.493433 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:24.493433 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9 [sys.catalog]: This master's current role is: LEADER
W20260620 04:30:24.497442 3402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:30:24.497524 3402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:30:24.497582 3398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:24.498435 3398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:24.504321 3398 catalog_manager.cc:1347] Generated new cluster ID: 79999aab40a74e62b8e95b4c509de065
I20260620 04:30:24.504396 3398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:24.508788 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:24.504185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.0.98.62:38265)
0620 04:30:24.504327 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:24.504332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:24.504586 (+ 254us) server_negotiation.cc:407] Connection header received
0620 04:30:24.504715 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:24.504718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:24.504765 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:24.504846 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:24.505471 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:24.506012 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:24.506832 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:24.507065 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:24.507149 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:24.507928 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:24.507932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:24.507934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:24.508369 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:24.508398 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:24.508413 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:24.508488 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:24.508587 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 04:30:24.508648 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:30:24.561822 3398 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:24.562858 3398 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:24.567058 3398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a513483baa4a7da25eb7ed8daef5c9: Generated new TSK 0
I20260620 04:30:24.567728 3398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:30:24.579941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 3313
2026-06-20T04:30:24Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1362 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:24 dist-test-slave-rzqm krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:24 dist-test-slave-rzqm krb5kdc[3414](info): set up 2 sockets
Jun 20 04:30:24 dist-test-slave-rzqm krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260620 04:30:25.528833 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:25 dist-test-slave-rzqm krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:25.652802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35887
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:25.762732 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:25.763051 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:25.763109 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:25.767199 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:25.767270 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:25.767297 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:25.767318 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:25.767338 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:25.773119 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:25.774245 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:25.775265 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:25.783761 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:25.783926 3427 server_base.cc:1034] running on GCE node
W20260620 04:30:25.783764 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:25.783761 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:25.784646 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:25.785827 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:25.787007 3427 hybrid_clock.cc:648] HybridClock initialized: now 1781929825786989 us; error 34 us; skew 500 ppm
Jun 20 04:30:25 dist-test-slave-rzqm krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:25.789986 3427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:25.790903 3427 webserver.cc:460] Webserver started at http://127.0.98.62:41887/ using document root <none> and password file <none>
I20260620 04:30:25.791447 3427 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:25.791558 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:25.791765 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:25.793919 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "3b01a92b1cea41d18b5b2f82e5e65a79"
format_stamp: "Formatted at 2026-06-20 04:30:25 on dist-test-slave-rzqm"
server_key: "bb0c7ffea50a1bbb2da1c988d188efe2"
server_key_iv: "2e0bc77e5af18650fc38ff090a7ce650"
server_key_version: "encryptionkey@0"
I20260620 04:30:25.794461 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b01a92b1cea41d18b5b2f82e5e65a79"
format_stamp: "Formatted at 2026-06-20 04:30:25 on dist-test-slave-rzqm"
server_key: "bb0c7ffea50a1bbb2da1c988d188efe2"
server_key_iv: "2e0bc77e5af18650fc38ff090a7ce650"
server_key_version: "encryptionkey@0"
I20260620 04:30:25.798432 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:25.801132 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:25.802632 3427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:30:25.802778 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "3b01a92b1cea41d18b5b2f82e5e65a79"
format_stamp: "Formatted at 2026-06-20 04:30:25 on dist-test-slave-rzqm"
server_key: "bb0c7ffea50a1bbb2da1c988d188efe2"
server_key_iv: "2e0bc77e5af18650fc38ff090a7ce650"
server_key_version: "encryptionkey@0"
I20260620 04:30:25.802910 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:25.827104 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:25.830513 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:25.830698 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:25.837783 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35887
I20260620 04:30:25.838047 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35887 every 8 connection(s)
I20260620 04:30:25.838781 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:25.839128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 3427
I20260620 04:30:25.839236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:25.839543 392 external_mini_cluster.cc:1428] Setting key 912655d48f203191078be3a2fba2c5c8
I20260620 04:30:25.842219 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:25.848156 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79: Bootstrap starting.
Jun 20 04:30:25 dist-test-slave-rzqm krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:25.850672 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:25.851395 3495 log.cc:824] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:25.853466 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79: No bootstrap required, opened a new log
I20260620 04:30:25.853777 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:25.841036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.0.98.62:35887)
0620 04:30:25.841535 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:25.841559 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:25.841598 (+ 39us) server_negotiation.cc:407] Connection header received
0620 04:30:25.842277 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:25.842300 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:25.842722 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:25.843074 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:25.843898 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:25.844947 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:25.845803 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:25.846280 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:25.846430 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:25.849139 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:25.849144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:25.849159 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:25.849188 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:25.850880 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:25.851340 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:25.851345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:25.851351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:25.851470 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:25.851741 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:25.851744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:25.851745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:25.852167 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:25.852370 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:25.852668 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 04:30:25.852956 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":164,"threads_started":1}
I20260620 04:30:25.857434 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b01a92b1cea41d18b5b2f82e5e65a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35887 } }
I20260620 04:30:25.857689 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:25.857761 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b01a92b1cea41d18b5b2f82e5e65a79, State: Initialized, Role: FOLLOWER
I20260620 04:30:25.858189 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b01a92b1cea41d18b5b2f82e5e65a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35887 } }
I20260620 04:30:25.858321 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:25.858385 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:25.858484 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:25.859433 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b01a92b1cea41d18b5b2f82e5e65a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35887 } }
I20260620 04:30:25.859799 3495 leader_election.cc:302] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b01a92b1cea41d18b5b2f82e5e65a79; no voters:
I20260620 04:30:25.860137 3495 leader_election.cc:288] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:25.860236 3500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:25.860467 3500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [term 1 LEADER]: Becoming Leader. State: Replica: 3b01a92b1cea41d18b5b2f82e5e65a79, State: Running, Role: LEADER
I20260620 04:30:25.860803 3500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b01a92b1cea41d18b5b2f82e5e65a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35887 } }
I20260620 04:30:25.861502 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:25.862619 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b01a92b1cea41d18b5b2f82e5e65a79. Latest consensus state: current_term: 1 leader_uuid: "3b01a92b1cea41d18b5b2f82e5e65a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b01a92b1cea41d18b5b2f82e5e65a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35887 } } }
I20260620 04:30:25.862617 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b01a92b1cea41d18b5b2f82e5e65a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b01a92b1cea41d18b5b2f82e5e65a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35887 } } }
I20260620 04:30:25.862753 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:25.862778 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:25.863148 3509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:25.865994 3509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:25.871697 3509 catalog_manager.cc:1347] Generated new cluster ID: fc433b4058ca42d3a72f66621f1a711b
I20260620 04:30:25.871784 3509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:25.895872 3509 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:25.897033 3509 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:25.902256 3509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b01a92b1cea41d18b5b2f82e5e65a79: Generated new TSK 0
I20260620 04:30:25.903152 3509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:30:25.931174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35887
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:26.043540 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:26.043840 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:26.043890 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:26.048178 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:26.048250 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:26.048317 3521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:26.053943 3521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:26.055096 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:26.056141 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:26.066325 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:26.066356 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:26.066452 3521 server_base.cc:1034] running on GCE node
W20260620 04:30:26.066329 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:26.067101 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:26.067763 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:26.068941 3521 hybrid_clock.cc:648] HybridClock initialized: now 1781929826068916 us; error 56 us; skew 500 ppm
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:26.075170 3521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:30:26.076159 3521 webserver.cc:460] Webserver started at http://127.0.98.1:40789/ using document root <none> and password file <none>
I20260620 04:30:26.076730 3521 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:26.076799 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:26.076992 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:26.079118 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "31cfc4cb32af4fb48224f4124494a44a"
format_stamp: "Formatted at 2026-06-20 04:30:26 on dist-test-slave-rzqm"
server_key: "4c5afe5deb0ba96791019299ed621430"
server_key_iv: "442da33d2f158bb4e75d97ed371b1178"
server_key_version: "encryptionkey@0"
I20260620 04:30:26.079671 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "31cfc4cb32af4fb48224f4124494a44a"
format_stamp: "Formatted at 2026-06-20 04:30:26 on dist-test-slave-rzqm"
server_key: "4c5afe5deb0ba96791019299ed621430"
server_key_iv: "442da33d2f158bb4e75d97ed371b1178"
server_key_version: "encryptionkey@0"
I20260620 04:30:26.083600 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 04:30:26.086133 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:26.087528 3521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:26.087671 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "31cfc4cb32af4fb48224f4124494a44a"
format_stamp: "Formatted at 2026-06-20 04:30:26 on dist-test-slave-rzqm"
server_key: "4c5afe5deb0ba96791019299ed621430"
server_key_iv: "442da33d2f158bb4e75d97ed371b1178"
server_key_version: "encryptionkey@0"
I20260620 04:30:26.087778 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:26.099798 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:26.102972 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:26.103166 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:26.103773 3521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:26.104625 3521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:26.104674 3521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:26.104727 3521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:26.104756 3521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:26.113380 3521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46603
I20260620 04:30:26.113409 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46603 every 8 connection(s)
I20260620 04:30:26.114297 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:26.117832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 3521
I20260620 04:30:26.117929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:26.118196 392 external_mini_cluster.cc:1428] Setting key 6670d477c121834dbb2bb8b3c7483e1a
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:30:26.129757 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:26.116066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45721 (local address 127.0.98.62:35887)
0620 04:30:26.116197 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:26.116200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:26.117068 (+ 868us) server_negotiation.cc:407] Connection header received
0620 04:30:26.118545 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:26.118549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:26.118612 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:26.118689 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:26.119908 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.120434 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.122023 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.122283 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.122408 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:26.125347 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:26.125352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:26.125355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:26.125390 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:26.127251 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:26.127905 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:26.127910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:26.127912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:26.128021 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:26.128436 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:26.128439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:26.128440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:26.128714 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:26.128806 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:26.129437 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 04:30:26.129626 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:30:26.130663 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:26.116463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35887 (local address 127.0.98.1:45721)
0620 04:30:26.116896 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:26.116939 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:26.118321 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:26.118879 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:26.118887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:26.119251 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:26.119761 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:26.119775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.120564 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.120567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:26.121863 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:26.121879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.122466 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.122469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:26.122646 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:26.123272 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:26.123290 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:26.125119 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:26.127411 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:26.127418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:26.127435 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:26.127772 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:26.128125 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:26.128148 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:26.128149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:26.128321 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:26.128829 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:26.128838 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:26.129165 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 04:30:26.129346 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 04:30:26.129678 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
I20260620 04:30:26.132874 3650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35887
I20260620 04:30:26.133178 3650 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:26.133826 3650 heartbeater.cc:511] Master 127.0.98.62:35887 requested a full tablet report, sending...
I20260620 04:30:26.135671 3459 ts_manager.cc:194] Registered new tserver with Master: 31cfc4cb32af4fb48224f4124494a44a (127.0.98.1:46603)
I20260620 04:30:26.136839 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:30:26.148087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35887
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:26.277992 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:26.278278 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:26.278321 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:26.282807 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:26.282904 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:26.282968 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:26.288661 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:26.289835 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:26.290899 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:26.301350 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:26.301353 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:26.301353 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:26.301397 3655 server_base.cc:1034] running on GCE node
I20260620 04:30:26.302199 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:26.302933 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:26.304180 3655 hybrid_clock.cc:648] HybridClock initialized: now 1781929826304159 us; error 39 us; skew 500 ppm
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:26.307243 3655 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:30:26.308279 3655 webserver.cc:460] Webserver started at http://127.0.98.2:44121/ using document root <none> and password file <none>
I20260620 04:30:26.308857 3655 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:26.308928 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:26.309180 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:26.311343 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "44f773bc247a4dadb0ddf49abcb13905"
format_stamp: "Formatted at 2026-06-20 04:30:26 on dist-test-slave-rzqm"
server_key: "4db150c0116fa8fb177070a84a3397dc"
server_key_iv: "a77741d0279f303efb7f9b265b4445f5"
server_key_version: "encryptionkey@0"
I20260620 04:30:26.311960 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44f773bc247a4dadb0ddf49abcb13905"
format_stamp: "Formatted at 2026-06-20 04:30:26 on dist-test-slave-rzqm"
server_key: "4db150c0116fa8fb177070a84a3397dc"
server_key_iv: "a77741d0279f303efb7f9b265b4445f5"
server_key_version: "encryptionkey@0"
I20260620 04:30:26.316133 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 04:30:26.318986 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:26.320389 3655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:26.320535 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "44f773bc247a4dadb0ddf49abcb13905"
format_stamp: "Formatted at 2026-06-20 04:30:26 on dist-test-slave-rzqm"
server_key: "4db150c0116fa8fb177070a84a3397dc"
server_key_iv: "a77741d0279f303efb7f9b265b4445f5"
server_key_version: "encryptionkey@0"
I20260620 04:30:26.320660 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:26.366906 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:26.370287 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:26.370476 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:26.371094 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:26.371944 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:26.371994 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:26.372046 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:26.372076 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:26.381054 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44001
I20260620 04:30:26.381083 3783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44001 every 8 connection(s)
I20260620 04:30:26.382092 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:26.385447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 3655
I20260620 04:30:26.385563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:26.385864 392 external_mini_cluster.cc:1428] Setting key 679b7aea3b4582d13d5a5a826019bdf6
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:30:26.398061 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:26.384081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53797 (local address 127.0.98.62:35887)
0620 04:30:26.384202 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:26.384206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:26.385077 (+ 871us) server_negotiation.cc:407] Connection header received
0620 04:30:26.386716 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:26.386719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:26.386763 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:26.386833 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:26.388440 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.389071 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.390610 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.390980 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.391133 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:26.394287 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:26.394290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:26.394292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:26.394320 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:26.395716 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:26.396228 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:26.396231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:26.396232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:26.396318 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:26.396673 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:26.396675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:26.396676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:26.396865 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:26.396942 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:26.397680 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 04:30:26.397820 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:30:26.398933 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:26.384436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35887 (local address 127.0.98.2:53797)
0620 04:30:26.384901 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:26.384947 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:26.386498 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:26.387028 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:26.387037 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:26.387563 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:26.388274 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:26.388294 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.389204 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.389208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:26.390463 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:26.390473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.391207 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.391211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:26.391385 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:26.392135 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:26.392158 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:26.394058 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:26.395835 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:26.395840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:26.395855 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:26.396113 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:26.396424 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:26.396428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:26.396430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:26.396588 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:26.396967 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:26.396973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:26.397413 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 04:30:26.397602 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 04:30:26.397973 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260620 04:30:26.400416 3784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35887
I20260620 04:30:26.400689 3784 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:26.401398 3784 heartbeater.cc:511] Master 127.0.98.62:35887 requested a full tablet report, sending...
I20260620 04:30:26.402702 3459 ts_manager.cc:194] Registered new tserver with Master: 44f773bc247a4dadb0ddf49abcb13905 (127.0.98.2:44001)
I20260620 04:30:26.403493 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:30:26.416412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35887
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:26.531854 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:26.532126 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:26.532171 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:26.536441 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:26.536525 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:26.536604 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:26.542086 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:26.543139 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:26.544123 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:26.554024 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:26.554018 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:26.554013 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:26.554121 3789 server_base.cc:1034] running on GCE node
I20260620 04:30:26.554654 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:26.555341 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:26.556535 3789 hybrid_clock.cc:648] HybridClock initialized: now 1781929826556508 us; error 40 us; skew 500 ppm
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:26.559629 3789 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:30:26.560665 3789 webserver.cc:460] Webserver started at http://127.0.98.3:36599/ using document root <none> and password file <none>
I20260620 04:30:26.561343 3789 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:26.561425 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:26.561643 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:26.564088 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "68c5326e1ee54a439183e8845bba80dc"
format_stamp: "Formatted at 2026-06-20 04:30:26 on dist-test-slave-rzqm"
server_key: "8294bc089b4413d12be75e3ec171e5d5"
server_key_iv: "11b32f2df377cd5e773e3125b93763e7"
server_key_version: "encryptionkey@0"
I20260620 04:30:26.564705 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "68c5326e1ee54a439183e8845bba80dc"
format_stamp: "Formatted at 2026-06-20 04:30:26 on dist-test-slave-rzqm"
server_key: "8294bc089b4413d12be75e3ec171e5d5"
server_key_iv: "11b32f2df377cd5e773e3125b93763e7"
server_key_version: "encryptionkey@0"
I20260620 04:30:26.568768 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:30:26.571491 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:26.572768 3789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:26.572890 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "68c5326e1ee54a439183e8845bba80dc"
format_stamp: "Formatted at 2026-06-20 04:30:26 on dist-test-slave-rzqm"
server_key: "8294bc089b4413d12be75e3ec171e5d5"
server_key_iv: "11b32f2df377cd5e773e3125b93763e7"
server_key_version: "encryptionkey@0"
I20260620 04:30:26.573036 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:26.603590 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:26.606961 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:26.607149 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:26.607760 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:26.608604 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:26.608652 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:26.608705 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:26.608731 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:26.617761 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43737
I20260620 04:30:26.617794 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43737 every 8 connection(s)
I20260620 04:30:26.618863 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:26.623291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 3789
I20260620 04:30:26.623410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:26.623747 392 external_mini_cluster.cc:1428] Setting key a8be9622b16e39fb01cd7414eb5bcfff
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:26.632874 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:26.621144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35501 (local address 127.0.98.62:35887)
0620 04:30:26.622696 (+ 1552us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:26.622700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:26.622716 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:30:26.623672 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:26.623676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:26.623732 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:26.623827 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:26.625359 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.625891 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.627008 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.627227 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.627313 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:26.629651 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:26.629654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:26.629656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:26.629681 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:26.630936 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:26.631541 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:26.631544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:26.631545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:26.631626 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:26.631962 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:26.631964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:26.631965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:26.632131 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:26.632198 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:26.632655 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 04:30:26.632753 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1466}
I20260620 04:30:26.633775 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:26.621704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35887 (local address 127.0.98.3:35501)
0620 04:30:26.622160 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:26.622206 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:26.623438 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:26.624048 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:26.624056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:26.624588 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:26.625225 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:26.625245 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.626002 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.626005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:26.626898 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:26.626907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.627356 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.627360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:26.627530 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:26.628023 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:26.628039 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:26.629471 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:26.631085 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:26.631091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:26.631106 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:26.631400 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:26.631739 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:26.631741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:26.631743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:26.631885 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:26.632202 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:26.632207 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:26.632449 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 04:30:26.632608 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 04:30:26.632902 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260620 04:30:26.634846 3918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35887
I20260620 04:30:26.635092 3918 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:26.635601 3918 heartbeater.cc:511] Master 127.0.98.62:35887 requested a full tablet report, sending...
I20260620 04:30:26.636634 3458 ts_manager.cc:194] Registered new tserver with Master: 68c5326e1ee54a439183e8845bba80dc (127.0.98.3:43737)
I20260620 04:30:26.637338 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:30:26.637408 3458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35501
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929826, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929826, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:26.661648 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:26.653230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.0.98.62:35887)
0620 04:30:26.653415 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:26.653420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:26.653635 (+ 215us) server_negotiation.cc:407] Connection header received
0620 04:30:26.653791 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:26.653794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:26.653838 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:26.653900 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:26.654628 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.655177 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.656030 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:26.656266 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:26.656360 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:26.658776 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:26.658779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:26.658781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:26.658809 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:26.660179 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:26.660512 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:26.660515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:26.660516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:26.660585 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:26.660778 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:26.660780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:26.660781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:26.660988 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:26.661156 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:26.661398 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 04:30:26.661515 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 04:30:26.662037 3458 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46374
I20260620 04:30:26.662856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 3521
I20260620 04:30:26.667847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 3655
I20260620 04:30:26.672511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 3789
I20260620 04:30:26.676673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 3427
2026-06-20T04:30:26Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2098 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3930](info): set up 2 sockets
Jun 20 04:30:26 dist-test-slave-rzqm krb5kdc[3930](info): commencing operation
krb5kdc: starting...
W20260620 04:30:27.632947 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:27 dist-test-slave-rzqm krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:27.751992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46721
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:27.866447 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:27.866750 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:27.866792 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:27.870862 3943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:27.870929 3943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:27.870946 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:27.870961 3943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:27.870975 3943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:27.876703 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:27.877910 3943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:27.878938 3943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:27.887887 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:27.887866 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:27.887934 3943 server_base.cc:1034] running on GCE node
W20260620 04:30:27.887894 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:27.888710 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:27.889953 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:27.891140 3943 hybrid_clock.cc:648] HybridClock initialized: now 1781929827891120 us; error 33 us; skew 500 ppm
Jun 20 04:30:27 dist-test-slave-rzqm krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:27.899530 3943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:27.900588 3943 webserver.cc:460] Webserver started at http://127.0.98.62:44057/ using document root <none> and password file <none>
I20260620 04:30:27.901196 3943 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:27.901263 3943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:27.901459 3943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:27.903620 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "03180ae88b1e4a82ba727252fdb5bf0c"
format_stamp: "Formatted at 2026-06-20 04:30:27 on dist-test-slave-rzqm"
server_key: "92ae353ab17dfdf57ce1a788eae8f45d"
server_key_iv: "5b9e8dda4e4768b3df95a54468a85cab"
server_key_version: "encryptionkey@0"
I20260620 04:30:27.904186 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "03180ae88b1e4a82ba727252fdb5bf0c"
format_stamp: "Formatted at 2026-06-20 04:30:27 on dist-test-slave-rzqm"
server_key: "92ae353ab17dfdf57ce1a788eae8f45d"
server_key_iv: "5b9e8dda4e4768b3df95a54468a85cab"
server_key_version: "encryptionkey@0"
I20260620 04:30:27.908203 3943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:30:27.910858 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:27.912137 3943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:27.912253 3943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "03180ae88b1e4a82ba727252fdb5bf0c"
format_stamp: "Formatted at 2026-06-20 04:30:27 on dist-test-slave-rzqm"
server_key: "92ae353ab17dfdf57ce1a788eae8f45d"
server_key_iv: "5b9e8dda4e4768b3df95a54468a85cab"
server_key_version: "encryptionkey@0"
I20260620 04:30:27.912374 3943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:27.936676 3943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:27.940420 3943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:27.940589 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:27.947728 3943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46721
I20260620 04:30:27.947748 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46721 every 8 connection(s)
I20260620 04:30:27.948784 3943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:27.951763 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:27.957502 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c: Bootstrap starting.
I20260620 04:30:27.959092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 3943
I20260620 04:30:27.959200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:27.959484 392 external_mini_cluster.cc:1428] Setting key b8841f109b57d7df56cb8da2c0c2de77
I20260620 04:30:27.960171 4011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:27.961131 4011 log.cc:824] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:27.963133 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c: No bootstrap required, opened a new log
I20260620 04:30:27.966683 4011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03180ae88b1e4a82ba727252fdb5bf0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46721 } }
I20260620 04:30:27.966921 4011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:27.966964 4011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03180ae88b1e4a82ba727252fdb5bf0c, State: Initialized, Role: FOLLOWER
I20260620 04:30:27.967432 4011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03180ae88b1e4a82ba727252fdb5bf0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46721 } }
I20260620 04:30:27.967569 4011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:27.967639 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:27.967746 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [term 0 FOLLOWER]: Advancing to term 1
Jun 20 04:30:27 dist-test-slave-rzqm krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:27.968737 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03180ae88b1e4a82ba727252fdb5bf0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46721 } }
I20260620 04:30:27.969144 4011 leader_election.cc:302] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03180ae88b1e4a82ba727252fdb5bf0c; no voters:
I20260620 04:30:27.969529 4011 leader_election.cc:288] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:27.969637 4016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:27.969856 4016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [term 1 LEADER]: Becoming Leader. State: Replica: 03180ae88b1e4a82ba727252fdb5bf0c, State: Running, Role: LEADER
I20260620 04:30:27.970209 4016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03180ae88b1e4a82ba727252fdb5bf0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46721 } }
I20260620 04:30:27.970973 4011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:27.971403 4016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03180ae88b1e4a82ba727252fdb5bf0c. Latest consensus state: current_term: 1 leader_uuid: "03180ae88b1e4a82ba727252fdb5bf0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03180ae88b1e4a82ba727252fdb5bf0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46721 } } }
I20260620 04:30:27.971607 4016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:27.971807 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03180ae88b1e4a82ba727252fdb5bf0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03180ae88b1e4a82ba727252fdb5bf0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46721 } } }
I20260620 04:30:27.971942 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:27.972215 4021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:27.974717 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:27.960659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.0.98.62:46721)
0620 04:30:27.961165 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:27.961182 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:27.961224 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:30:27.962025 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:27.962047 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:27.962398 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:27.962724 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:27.963540 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:27.964554 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:27.965464 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:27.965954 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:27.966103 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:27.968580 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:27.968586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:27.968601 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:27.968629 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:27.970714 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:27.971242 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:27.971249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:27.971256 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:27.971407 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:27.971754 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:27.971757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:27.971759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:27.972331 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:27.973238 (+ 907us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:27.973393 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 04:30:27.973727 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":146,"threads_started":1}
I20260620 04:30:27.975282 4021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:27.980756 4021 catalog_manager.cc:1347] Generated new cluster ID: 3747159c875b44b2ad421aa9aecc6b17
I20260620 04:30:27.980850 4021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:28.006002 4021 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:28.007043 4021 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:28.015089 4021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03180ae88b1e4a82ba727252fdb5bf0c: Generated new TSK 0
I20260620 04:30:28.016067 4021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260620 04:30:28.312896 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.289s user 0.000s sys 0.003s
I20260620 04:30:28.313417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46721
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:28.424299 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:28.424603 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:28.424666 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:28.428812 4037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:28.428892 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:28.428978 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:28.434386 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:28.435508 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:28.436558 4037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:28.446583 4042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:28.446597 4045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:28.446597 4043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:28.446832 4037 server_base.cc:1034] running on GCE node
I20260620 04:30:28.447257 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:28.447935 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:28.449098 4037 hybrid_clock.cc:648] HybridClock initialized: now 1781929828449060 us; error 49 us; skew 500 ppm
Jun 20 04:30:28 dist-test-slave-rzqm krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:28.451862 4037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:30:28.452730 4037 webserver.cc:460] Webserver started at http://127.0.98.1:33123/ using document root <none> and password file <none>
I20260620 04:30:28.453310 4037 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:28.453373 4037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:28.453545 4037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:28.455619 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "8ef7ff77dfca463f9f6ffe764479f17e"
format_stamp: "Formatted at 2026-06-20 04:30:28 on dist-test-slave-rzqm"
server_key: "35946816596722a24491dee95e4149b9"
server_key_iv: "f51348d2de31d93e8d8c9731710aca13"
server_key_version: "encryptionkey@0"
I20260620 04:30:28.456158 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8ef7ff77dfca463f9f6ffe764479f17e"
format_stamp: "Formatted at 2026-06-20 04:30:28 on dist-test-slave-rzqm"
server_key: "35946816596722a24491dee95e4149b9"
server_key_iv: "f51348d2de31d93e8d8c9731710aca13"
server_key_version: "encryptionkey@0"
I20260620 04:30:28.459982 4037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 04:30:28.462586 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:28.463812 4037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:28.463930 4037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "8ef7ff77dfca463f9f6ffe764479f17e"
format_stamp: "Formatted at 2026-06-20 04:30:28 on dist-test-slave-rzqm"
server_key: "35946816596722a24491dee95e4149b9"
server_key_iv: "f51348d2de31d93e8d8c9731710aca13"
server_key_version: "encryptionkey@0"
I20260620 04:30:28.464051 4037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:28.477804 4037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:28.481372 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:28.481570 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:28.482225 4037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:28.483098 4037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:28.483144 4037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:28.483230 4037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:28.483264 4037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:28.491982 4037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39957
I20260620 04:30:28.492012 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39957 every 8 connection(s)
I20260620 04:30:28.492964 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:28.499732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 4037
I20260620 04:30:28.499887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:28.500181 392 external_mini_cluster.cc:1428] Setting key 1fbe423c734d08886ebbf4c3746b6393
Jun 20 04:30:28 dist-test-slave-rzqm krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:28.507601 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:28.494900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34409 (local address 127.0.98.62:46721)
0620 04:30:28.495163 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:28.495168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:28.495941 (+ 773us) server_negotiation.cc:407] Connection header received
0620 04:30:28.497369 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:28.497373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:28.497423 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:28.497511 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:28.498750 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.499329 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.500852 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.501146 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.501234 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:28.504134 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:28.504137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:28.504139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:28.504164 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:28.505598 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:28.506101 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:28.506105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:28.506107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:28.506183 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:28.506503 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:28.506506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:28.506508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:28.506695 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:28.506781 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:28.507293 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 04:30:28.507422 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":56,"threads_started":1}
I20260620 04:30:28.508463 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:28.495291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46721 (local address 127.0.98.1:34409)
0620 04:30:28.495775 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:28.495826 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:28.497129 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:28.497713 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:28.497721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:28.498083 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:28.498590 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:28.498603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.499463 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.499467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:28.500747 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:28.500756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.501323 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.501328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:28.501494 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:28.502093 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:28.502116 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:28.503933 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:28.505710 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:28.505717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:28.505736 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:28.506006 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:28.506265 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:28.506268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:28.506271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:28.506418 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:28.506793 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:28.506800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:28.507058 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 04:30:28.507239 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 04:30:28.507594 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:30:28.509809 4166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46721
I20260620 04:30:28.510089 4166 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:28.510584 4166 heartbeater.cc:511] Master 127.0.98.62:46721 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:30:28.512177 3975 ts_manager.cc:194] Registered new tserver with Master: 8ef7ff77dfca463f9f6ffe764479f17e (127.0.98.1:39957)
I20260620 04:30:28.513352 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:30:28.528427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46721
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:28.639267 4172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:28.639549 4172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:28.639592 4172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:28.643903 4172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:28.643987 4172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:28.644052 4172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:28.649528 4172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:28.650559 4172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:28.651535 4172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:28.660774 4180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:28.660831 4178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:28.661067 4172 server_base.cc:1034] running on GCE node
W20260620 04:30:28.660794 4177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:28.661532 4172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:28.662134 4172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:28.663292 4172 hybrid_clock.cc:648] HybridClock initialized: now 1781929828663275 us; error 29 us; skew 500 ppm
Jun 20 04:30:28 dist-test-slave-rzqm krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:28.666209 4172 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:30:28.667098 4172 webserver.cc:460] Webserver started at http://127.0.98.2:46873/ using document root <none> and password file <none>
I20260620 04:30:28.667688 4172 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:28.667753 4172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:28.667922 4172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:28.670084 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "3921e6beee1c43e38d579d3d848440ba"
format_stamp: "Formatted at 2026-06-20 04:30:28 on dist-test-slave-rzqm"
server_key: "c5de004ccd34d56750b0e4b4cfa6aa48"
server_key_iv: "e6f18e07e29fa239d4fc6e1a9cf0843c"
server_key_version: "encryptionkey@0"
I20260620 04:30:28.670640 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3921e6beee1c43e38d579d3d848440ba"
format_stamp: "Formatted at 2026-06-20 04:30:28 on dist-test-slave-rzqm"
server_key: "c5de004ccd34d56750b0e4b4cfa6aa48"
server_key_iv: "e6f18e07e29fa239d4fc6e1a9cf0843c"
server_key_version: "encryptionkey@0"
I20260620 04:30:28.674449 4172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:28.676862 4187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:28.678133 4172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:28.678237 4172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "3921e6beee1c43e38d579d3d848440ba"
format_stamp: "Formatted at 2026-06-20 04:30:28 on dist-test-slave-rzqm"
server_key: "c5de004ccd34d56750b0e4b4cfa6aa48"
server_key_iv: "e6f18e07e29fa239d4fc6e1a9cf0843c"
server_key_version: "encryptionkey@0"
I20260620 04:30:28.678350 4172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:28.696902 4172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:28.699877 4172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:28.700060 4172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:28.700682 4172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:28.701553 4172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:28.701612 4172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:28.701655 4172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:28.701683 4172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:28.710017 4172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42129
I20260620 04:30:28.710052 4300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42129 every 8 connection(s)
I20260620 04:30:28.710915 4172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:28.715155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 4172
I20260620 04:30:28.715286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:28.715588 392 external_mini_cluster.cc:1428] Setting key eff42a66e71eff4d7a9ace9ee58c8062
Jun 20 04:30:28 dist-test-slave-rzqm krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:30:28.725704 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:28.712723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58977 (local address 127.0.98.62:46721)
0620 04:30:28.712846 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:28.712850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:28.713724 (+ 874us) server_negotiation.cc:407] Connection header received
0620 04:30:28.715131 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:28.715135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:28.715194 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:28.715265 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:28.716681 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.717347 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.718875 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.719214 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.719321 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:28.722005 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:28.722008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:28.722010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:28.722038 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:28.723406 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:28.724017 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:28.724020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:28.724021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:28.724103 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:28.724508 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:28.724511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:28.724512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:28.724697 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:28.724814 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:28.725429 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 04:30:28.725545 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:30:28.726516 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:28.713121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46721 (local address 127.0.98.2:58977)
0620 04:30:28.713571 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:28.713615 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:28.714884 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:28.715548 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:28.715557 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:28.716038 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:28.716549 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:28.716563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.717477 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.717480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:28.718757 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:28.718768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.719388 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.719392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:28.719545 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:28.720122 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:28.720146 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:28.721790 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:28.723530 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:28.723537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:28.723556 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:28.723908 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:28.724205 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:28.724208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:28.724210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:28.724400 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:28.724820 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:28.724827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:28.725187 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 04:30:28.725357 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 04:30:28.725645 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:30:28.727753 4301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46721
I20260620 04:30:28.728031 4301 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:28.728623 4301 heartbeater.cc:511] Master 127.0.98.62:46721 requested a full tablet report, sending...
I20260620 04:30:28.729755 3975 ts_manager.cc:194] Registered new tserver with Master: 3921e6beee1c43e38d579d3d848440ba (127.0.98.2:42129)
I20260620 04:30:28.730337 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:30:28.742460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46721
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:28.853945 4306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:28.854239 4306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:28.854303 4306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:28.858692 4306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:28.858767 4306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:28.858850 4306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:28.864385 4306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:28.865588 4306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:28.866617 4306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:28.876227 4314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:28.876259 4312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:28.876387 4306 server_base.cc:1034] running on GCE node
W20260620 04:30:28.876282 4311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:28.876958 4306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:28.877624 4306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:28.878780 4306 hybrid_clock.cc:648] HybridClock initialized: now 1781929828878731 us; error 62 us; skew 500 ppm
Jun 20 04:30:28 dist-test-slave-rzqm krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:28.881546 4306 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:30:28.882515 4306 webserver.cc:460] Webserver started at http://127.0.98.3:45725/ using document root <none> and password file <none>
I20260620 04:30:28.883064 4306 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:28.883126 4306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:28.883316 4306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:28.885522 4306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "7ebb2ed2201f45b4ba343c5bae486866"
format_stamp: "Formatted at 2026-06-20 04:30:28 on dist-test-slave-rzqm"
server_key: "322921a0e7776444d10856d776cf04dd"
server_key_iv: "c908728386af93e0028b51593f219947"
server_key_version: "encryptionkey@0"
I20260620 04:30:28.886067 4306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7ebb2ed2201f45b4ba343c5bae486866"
format_stamp: "Formatted at 2026-06-20 04:30:28 on dist-test-slave-rzqm"
server_key: "322921a0e7776444d10856d776cf04dd"
server_key_iv: "c908728386af93e0028b51593f219947"
server_key_version: "encryptionkey@0"
I20260620 04:30:28.889791 4306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:28.892266 4321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:28.893607 4306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:28.893720 4306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "7ebb2ed2201f45b4ba343c5bae486866"
format_stamp: "Formatted at 2026-06-20 04:30:28 on dist-test-slave-rzqm"
server_key: "322921a0e7776444d10856d776cf04dd"
server_key_iv: "c908728386af93e0028b51593f219947"
server_key_version: "encryptionkey@0"
I20260620 04:30:28.893819 4306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:28.909657 4306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:28.912811 4306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:28.913035 4306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:28.913652 4306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:28.914476 4306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:28.914525 4306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:28.914577 4306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:28.914606 4306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:28.923193 4306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42249
I20260620 04:30:28.923275 4434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42249 every 8 connection(s)
I20260620 04:30:28.924166 4306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:28.928942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 4306
I20260620 04:30:28.929088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:28.929342 392 external_mini_cluster.cc:1428] Setting key 18030b8acd5d4e6efb227cfd5ce52ef7
Jun 20 04:30:28 dist-test-slave-rzqm krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:28.937912 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:28.926059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58353 (local address 127.0.98.62:46721)
0620 04:30:28.926179 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:28.926183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:28.927114 (+ 931us) server_negotiation.cc:407] Connection header received
0620 04:30:28.928487 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:28.928491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:28.928548 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:28.928618 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:28.929996 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.930680 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.931933 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.932152 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.932248 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:28.934753 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:28.934757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:28.934759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:28.934783 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:28.935967 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:28.936432 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:28.936435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:28.936436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:28.936512 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:28.936833 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:28.936836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:28.936837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:28.937079 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:28.937168 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:28.937667 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 04:30:28.937778 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:30:28.938768 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:28.926455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46721 (local address 127.0.98.3:58353)
0620 04:30:28.926951 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:28.927005 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:28.928260 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:28.928810 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:28.928817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:28.929264 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:28.929841 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:28.929862 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.930780 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.930783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:28.931816 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:28.931825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.932279 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.932282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:28.932417 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:28.932990 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:28.933037 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:28.934576 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:28.936069 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:28.936075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:28.936088 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:28.936340 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:28.936580 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:28.936582 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:28.936584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:28.936717 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:28.937166 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:28.937171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:28.937420 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 04:30:28.937591 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 04:30:28.937851 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20260620 04:30:28.939990 4435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46721
I20260620 04:30:28.940227 4435 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:28.940727 4435 heartbeater.cc:511] Master 127.0.98.62:46721 requested a full tablet report, sending...
I20260620 04:30:28.941937 3975 ts_manager.cc:194] Registered new tserver with Master: 7ebb2ed2201f45b4ba343c5bae486866 (127.0.98.3:42249)
I20260620 04:30:28.942620 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58353
I20260620 04:30:28.943029 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 04:30:28 dist-test-slave-rzqm krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:30:28 dist-test-slave-rzqm krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:30:28.969880 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:28.959957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.0.98.1:39957)
0620 04:30:28.960314 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:28.960325 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:28.960345 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:30:28.960389 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:28.960393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:28.960606 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:28.960736 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:28.961496 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.962283 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.963140 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.963526 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.963625 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:28.966130 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:28.966140 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:28.966153 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:28.966183 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:28.967841 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:28.968324 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:28.968332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:28.968334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:28.968428 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:28.968796 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:28.968799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:28.968800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:28.969116 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:28.969298 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:28.969533 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 04:30:28.969678 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":132,"threads_started":1}
Jun 20 04:30:28 dist-test-slave-rzqm krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:28.981853 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:28.972922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.0.98.62:46721)
0620 04:30:28.973076 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:28.973081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:28.973190 (+ 109us) server_negotiation.cc:407] Connection header received
0620 04:30:28.973282 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:28.973284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:28.973323 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:28.973388 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:28.974052 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.974586 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.975446 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:28.975682 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:28.975769 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:28.978541 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:28.978545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:28.978547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:28.978577 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:28.980046 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:28.980528 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:28.980532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:28.980533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:28.980615 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:28.980870 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:28.980873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:28.980875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:28.981171 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:28.981295 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:28.981605 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 04:30:28.981721 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 04:30:28.982264 3975 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39726
I20260620 04:30:28.982980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 4037
I20260620 04:30:28.987706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 4172
I20260620 04:30:28.992553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 4306
I20260620 04:30:28.997129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 3943
2026-06-20T04:30:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2320 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:29 dist-test-slave-rzqm krb5kdc[4447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:29 dist-test-slave-rzqm krb5kdc[4447](info): set up 2 sockets
Jun 20 04:30:29 dist-test-slave-rzqm krb5kdc[4447](info): commencing operation
krb5kdc: starting...
W20260620 04:30:29.960929 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:30 dist-test-slave-rzqm krb5kdc[4447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:30.077781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39595
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:30.191087 4460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:30.191388 4460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:30.191448 4460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:30.195636 4460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:30.195722 4460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:30.195750 4460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:30.195771 4460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:30.195793 4460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:30.201438 4460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:30.202585 4460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:30.203637 4460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:30.211956 4465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:30.211956 4468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:30.211977 4466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:30.212321 4460 server_base.cc:1034] running on GCE node
I20260620 04:30:30.212949 4460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:30.214131 4460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:30.215303 4460 hybrid_clock.cc:648] HybridClock initialized: now 1781929830215283 us; error 32 us; skew 500 ppm
Jun 20 04:30:30 dist-test-slave-rzqm krb5kdc[4447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:30.218092 4460 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:30.219038 4460 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:30.219129 4460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:30.219333 4460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:30.221529 4460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "e867f8a978614ecea8147a970934c0ff"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "88984013c4df38e8abe68e0466468cdf"
server_key_iv: "18200fb0de53a001010e90f0c462ec27"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.222100 4460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "e867f8a978614ecea8147a970934c0ff"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "88984013c4df38e8abe68e0466468cdf"
server_key_iv: "18200fb0de53a001010e90f0c462ec27"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.226074 4460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:30:30.228644 4474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:30.230007 4460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:30:30.230173 4460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "e867f8a978614ecea8147a970934c0ff"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "88984013c4df38e8abe68e0466468cdf"
server_key_iv: "18200fb0de53a001010e90f0c462ec27"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.230284 4460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:30.240695 4460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:30.244100 4460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:30.244300 4460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:30.251367 4460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39595
I20260620 04:30:30.251616 4526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39595 every 8 connection(s)
I20260620 04:30:30.252033 4460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:30.253954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 4460
I20260620 04:30:30.254091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:30.254361 392 external_mini_cluster.cc:1428] Setting key a2b26a39eef512c281cca42e4c6ca6f5
I20260620 04:30:30.255192 4527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:30.260867 4527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff: Bootstrap starting.
Jun 20 04:30:30 dist-test-slave-rzqm krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:30.263439 4527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:30.264335 4527 log.cc:824] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:30.266988 4527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff: No bootstrap required, opened a new log
I20260620 04:30:30.269604 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:30.255833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.0.98.62:39595)
0620 04:30:30.256296 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:30.256312 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:30.256349 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:30:30.257067 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:30.257091 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:30.257434 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:30.257758 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:30.258526 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.259505 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.260348 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.260822 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.260963 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:30.263639 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:30.263646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:30.263667 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:30.263705 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:30.266021 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:30.266581 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:30.266588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:30.266594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:30.266791 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:30.267091 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:30.267094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:30.267096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:30.267647 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:30.267840 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:30.268163 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 04:30:30.268482 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":150,"threads_started":1}
I20260620 04:30:30.271649 4527 raft_consensus.cc:348] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e867f8a978614ecea8147a970934c0ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39595 } }
I20260620 04:30:30.271921 4527 raft_consensus.cc:374] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:30.272038 4527 raft_consensus.cc:729] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e867f8a978614ecea8147a970934c0ff, State: Initialized, Role: FOLLOWER
I20260620 04:30:30.272478 4527 consensus_queue.cc:260] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e867f8a978614ecea8147a970934c0ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39595 } }
I20260620 04:30:30.272614 4527 raft_consensus.cc:388] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:30.272660 4527 raft_consensus.cc:482] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:30.272732 4527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:30.273702 4527 raft_consensus.cc:504] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e867f8a978614ecea8147a970934c0ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39595 } }
I20260620 04:30:30.274021 4527 leader_election.cc:302] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e867f8a978614ecea8147a970934c0ff; no voters:
I20260620 04:30:30.274293 4527 leader_election.cc:288] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:30.274420 4532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:30.274648 4532 raft_consensus.cc:686] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [term 1 LEADER]: Becoming Leader. State: Replica: e867f8a978614ecea8147a970934c0ff, State: Running, Role: LEADER
I20260620 04:30:30.275017 4532 consensus_queue.cc:237] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e867f8a978614ecea8147a970934c0ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39595 } }
I20260620 04:30:30.275568 4527 sys_catalog.cc:564] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:30.276638 4534 sys_catalog.cc:455] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e867f8a978614ecea8147a970934c0ff. Latest consensus state: current_term: 1 leader_uuid: "e867f8a978614ecea8147a970934c0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e867f8a978614ecea8147a970934c0ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39595 } } }
I20260620 04:30:30.276764 4534 sys_catalog.cc:458] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:30.277127 4533 sys_catalog.cc:455] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e867f8a978614ecea8147a970934c0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e867f8a978614ecea8147a970934c0ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39595 } } }
I20260620 04:30:30.277210 4533 sys_catalog.cc:458] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:30.277326 4541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:30.280371 4541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:30.287256 4541 catalog_manager.cc:1347] Generated new cluster ID: a7b337e6508b4d56a1733ee88b5e329d
I20260620 04:30:30.287350 4541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:30.309042 4541 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:30.310488 4541 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:30.321768 4541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff: Generated new TSK 0
I20260620 04:30:30.322566 4541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:30:30.355974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39595
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:30.470546 4553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:30.470849 4553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:30.470914 4553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:30.475035 4553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:30.475106 4553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:30.475189 4553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:30.480850 4553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:30.482048 4553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:30.483088 4553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:30.492319 4559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:30.492378 4561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:30.492378 4558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:30.492725 4553 server_base.cc:1034] running on GCE node
I20260620 04:30:30.493162 4553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:30.493782 4553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:30.494944 4553 hybrid_clock.cc:648] HybridClock initialized: now 1781929830494897 us; error 63 us; skew 500 ppm
Jun 20 04:30:30 dist-test-slave-rzqm krb5kdc[4447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:30.497733 4553 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:30:30.498464 4553 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:30.498534 4553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:30.498735 4553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:30.500792 4553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "dbcbc1a1a7a04008a99f42ec8e39361f"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "95b245d887181e3e84e2dd7e6ab92c8e"
server_key_iv: "8df1a822a33dfff067c781086429301d"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.501366 4553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dbcbc1a1a7a04008a99f42ec8e39361f"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "95b245d887181e3e84e2dd7e6ab92c8e"
server_key_iv: "8df1a822a33dfff067c781086429301d"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.505210 4553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:30:30.507718 4567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:30.509132 4553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:30.509261 4553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "dbcbc1a1a7a04008a99f42ec8e39361f"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "95b245d887181e3e84e2dd7e6ab92c8e"
server_key_iv: "8df1a822a33dfff067c781086429301d"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.509377 4553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:30.528517 4553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:30.531786 4553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:30.531973 4553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:30.532548 4553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:30.533495 4553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:30.533545 4553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:30.533614 4553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:30.533639 4553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:30.542167 4553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43617
I20260620 04:30:30.542191 4680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43617 every 8 connection(s)
I20260620 04:30:30.542879 4553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:30.543257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 4553
I20260620 04:30:30.543377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:30.543637 392 external_mini_cluster.cc:1428] Setting key bf986ff2ad323414aec8f754409306a4
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 20 04:30:30 dist-test-slave-rzqm krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:30.559731 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:30.544842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49499 (local address 127.0.98.62:39595)
0620 04:30:30.544976 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:30.544980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:30.546132 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 04:30:30.547716 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:30.547719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:30.547777 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:30.547857 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:30.549284 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.549820 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.550979 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.551214 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.551302 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:30.555830 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:30.555834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:30.555837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:30.555872 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:30.557538 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:30.558070 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:30.558074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:30.558076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:30.558182 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:30.558522 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:30.558525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:30.558527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:30.558768 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:30.558868 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:30.559395 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 04:30:30.559553 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:30:30.560529 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:30.545400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39595 (local address 127.0.98.1:49499)
0620 04:30:30.545936 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:30.545998 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:30.547476 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:30.548096 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:30.548105 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:30.548490 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:30.549135 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:30.549153 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.549947 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.549950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:30.550859 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:30.550868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.551363 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.551367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:30.551558 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:30.552218 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:30.552241 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:30.555600 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:30.557661 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:30.557667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:30.557682 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:30.557955 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:30.558270 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:30.558272 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:30.558273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:30.558427 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:30.558856 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:30.558862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:30.559122 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 04:30:30.559305 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 04:30:30.559597 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20260620 04:30:30.561878 4681 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39595
I20260620 04:30:30.562125 4681 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:30.562633 4681 heartbeater.cc:511] Master 127.0.98.62:39595 requested a full tablet report, sending...
I20260620 04:30:30.564177 4491 ts_manager.cc:194] Registered new tserver with Master: dbcbc1a1a7a04008a99f42ec8e39361f (127.0.98.1:43617)
I20260620 04:30:30.565436 4491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:30:30.570019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39595
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:30.679311 4686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:30.679586 4686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:30.679631 4686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:30.683656 4686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:30.683732 4686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:30.683796 4686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:30.689177 4686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:30.690235 4686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:30.691249 4686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:30.700484 4694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:30.700484 4691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:30.700544 4686 server_base.cc:1034] running on GCE node
W20260620 04:30:30.700485 4692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:30.701223 4686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:30.702100 4686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:30.703289 4686 hybrid_clock.cc:648] HybridClock initialized: now 1781929830703268 us; error 40 us; skew 500 ppm
Jun 20 04:30:30 dist-test-slave-rzqm krb5kdc[4447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:30.706313 4686 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:30:30.707100 4686 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:30.707175 4686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:30.707391 4686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:30.709606 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "64190df49e9e4636820ca97303f92a50"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "0b2e5a4ac25fda1c625f8cd6ed1433cb"
server_key_iv: "afa40781cbf7bab2187e54016dcd83fc"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.710201 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "64190df49e9e4636820ca97303f92a50"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "0b2e5a4ac25fda1c625f8cd6ed1433cb"
server_key_iv: "afa40781cbf7bab2187e54016dcd83fc"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.714367 4686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:30.717166 4700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:30.718516 4686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:30.718662 4686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "64190df49e9e4636820ca97303f92a50"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "0b2e5a4ac25fda1c625f8cd6ed1433cb"
server_key_iv: "afa40781cbf7bab2187e54016dcd83fc"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.718779 4686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:30.735862 4686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:30.739185 4686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:30.739341 4686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:30.739876 4686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:30.740656 4686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:30.740700 4686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:30.740737 4686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:30.740757 4686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:30.749297 4686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34031
I20260620 04:30:30.749337 4813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34031 every 8 connection(s)
I20260620 04:30:30.749898 4686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:30.756181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 4686
I20260620 04:30:30.756299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:30.756551 392 external_mini_cluster.cc:1428] Setting key 21047060e875f0364875a6fcc73e19e1
Jun 20 04:30:30 dist-test-slave-rzqm krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:30.763942 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:30.751762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46917 (local address 127.0.98.62:39595)
0620 04:30:30.751904 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:30.751908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:30.752762 (+ 854us) server_negotiation.cc:407] Connection header received
0620 04:30:30.754138 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:30.754142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:30.754197 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:30.754272 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:30.755462 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.756007 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.757306 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.757522 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.757609 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:30.760232 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:30.760236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:30.760239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:30.760267 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:30.761741 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:30.762281 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:30.762285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:30.762286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:30.762378 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:30.762779 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:30.762783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:30.762785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:30.763027 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:30.763137 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:30.763642 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 04:30:30.763776 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:30:30.764712 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:30.752148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39595 (local address 127.0.98.2:46917)
0620 04:30:30.752607 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:30.752651 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:30.753926 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:30.754454 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:30.754462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:30.754829 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:30.755318 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:30.755332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.756122 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.756125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:30.757192 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:30.757202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.757657 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.757661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:30.757865 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:30.758515 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:30.758531 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:30.760056 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:30.761865 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:30.761870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:30.761882 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:30.762188 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:30.762502 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:30.762505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:30.762507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:30.762650 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:30.763156 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:30.763162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:30.763409 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 04:30:30.763572 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 04:30:30.763855 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:30:30.765892 4814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39595
I20260620 04:30:30.766162 4814 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:30.766608 4814 heartbeater.cc:511] Master 127.0.98.62:39595 requested a full tablet report, sending...
I20260620 04:30:30.767652 4491 ts_manager.cc:194] Registered new tserver with Master: 64190df49e9e4636820ca97303f92a50 (127.0.98.2:34031)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:30:30.768368 4491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:30:30.781858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39595
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:30.894665 4819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:30.894964 4819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:30.895015 4819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:30.899140 4819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:30.899210 4819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:30.899273 4819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:30.904666 4819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:30.905805 4819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:30.906806 4819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:30.916123 4824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:30.916131 4827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:30.916173 4825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:30.916287 4819 server_base.cc:1034] running on GCE node
I20260620 04:30:30.916831 4819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:30.917518 4819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:30.918680 4819 hybrid_clock.cc:648] HybridClock initialized: now 1781929830918659 us; error 34 us; skew 500 ppm
Jun 20 04:30:30 dist-test-slave-rzqm krb5kdc[4447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:30.921654 4819 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:30:30.922405 4819 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:30.922472 4819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:30.922695 4819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:30.924841 4819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f75843c197744c2835829739f7673e8"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "91fb51e2b12ed622c4808ecdf8829371"
server_key_iv: "ce501b93535d1ac5d4d3cdb3ee463641"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.925439 4819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f75843c197744c2835829739f7673e8"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "91fb51e2b12ed622c4808ecdf8829371"
server_key_iv: "ce501b93535d1ac5d4d3cdb3ee463641"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.929235 4819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:30:30.931896 4833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:30.933329 4819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:30.933459 4819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "6f75843c197744c2835829739f7673e8"
format_stamp: "Formatted at 2026-06-20 04:30:30 on dist-test-slave-rzqm"
server_key: "91fb51e2b12ed622c4808ecdf8829371"
server_key_iv: "ce501b93535d1ac5d4d3cdb3ee463641"
server_key_version: "encryptionkey@0"
I20260620 04:30:30.933575 4819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:30.956580 4819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:30.959890 4819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:30.960071 4819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:30.960659 4819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:30.961539 4819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:30.961587 4819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:30.961650 4819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:30.961678 4819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:30.969743 4819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43919
I20260620 04:30:30.969765 4946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43919 every 8 connection(s)
I20260620 04:30:30.970359 4819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:30.978075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 4819
I20260620 04:30:30.978237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:30.978574 392 external_mini_cluster.cc:1428] Setting key bbd17bc89b04fc08eeaaa4e7d2a8b95b
Jun 20 04:30:30 dist-test-slave-rzqm krb5kdc[4447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:30.984735 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:30.972082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39147 (local address 127.0.98.62:39595)
0620 04:30:30.972224 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:30.972227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:30.973091 (+ 864us) server_negotiation.cc:407] Connection header received
0620 04:30:30.974479 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:30.974483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:30.974540 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:30.974613 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:30.975860 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.976453 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.977947 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.978186 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.978303 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:30.981393 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:30.981396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:30.981398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:30.981426 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:30.982758 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:30.983320 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:30.983323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:30.983324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:30.983407 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:30.983728 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:30.983730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:30.983731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:30.983911 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:30.983982 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:30.984496 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 04:30:30.984601 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:30:30.985652 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:30.972463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39595 (local address 127.0.98.3:39147)
0620 04:30:30.972928 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:30.972972 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:30.974260 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:30.974809 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:30.974816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:30.975182 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:30.975697 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:30.975712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.976586 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.976596 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:30.977780 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:30.977793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.978463 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.978468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:30.978741 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:30.979408 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:30.979426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:30.981219 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:30.982886 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:30.982893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:30.982907 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:30.983208 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:30.983485 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:30.983487 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:30.983488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:30.983643 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:30.983984 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:30.983989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:30.984290 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 04:30:30.984449 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 04:30:30.984724 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260620 04:30:30.986761 4947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39595
I20260620 04:30:30.987002 4947 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:30.987470 4947 heartbeater.cc:511] Master 127.0.98.62:39595 requested a full tablet report, sending...
I20260620 04:30:30.988528 4491 ts_manager.cc:194] Registered new tserver with Master: 6f75843c197744c2835829739f7673e8 (127.0.98.3:43919)
I20260620 04:30:30.989128 4491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39147
I20260620 04:30:30.992717 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:30:31.001350 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:30.994549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.0.98.62:39595)
0620 04:30:30.994655 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:30.994659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:30.994829 (+ 170us) server_negotiation.cc:407] Connection header received
0620 04:30:30.994941 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:30.994944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:30.994986 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:30.995046 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:30.995741 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.996247 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.997075 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:30.997287 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:30.997372 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:30.998624 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:30.998628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:30.998631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:30.998655 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:30.999767 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:31.000139 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:31.000142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:31.000144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:31.000219 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:31.000461 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:31.000465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:31.000466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:31.000710 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:31.000830 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:31.001090 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 04:30:31.001200 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 04:30:31.003971 4491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:31.006991 4491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:31.021103 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.016232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.0.98.3:43919)
0620 04:30:31.016612 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:31.016623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:31.016642 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:30:31.016756 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:31.016762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:31.016951 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:31.017413 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:31.017559 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.018383 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.019768 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.020429 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.020526 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.020675 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:31.020837 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 04:30:31.020929 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":107,"threads_started":1}
I20260620 04:30:31.021099 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.015468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43919 (local address 127.0.0.1:56178)
0620 04:30:31.016386 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:31.016399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:31.016481 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:31.017131 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:31.017134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:31.017150 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:31.017285 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:31.017293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.018533 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.018536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:31.019644 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:31.019653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.020540 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.020543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:31.020643 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.020689 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 04:30:31.020794 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 04:30:31.020917 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":57,"thread_start_us":111,"threads_started":1}
I20260620 04:30:31.021582 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.016125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43617 (local address 127.0.0.1:56206)
0620 04:30:31.016635 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:31.016645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:31.016740 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:31.017463 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:31.017466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:31.017481 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:31.017559 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:31.017565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.018866 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.018869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:31.019968 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:31.019976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.021091 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.021095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:31.021224 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.021294 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 04:30:31.021348 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 04:30:31.021403 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":46,"thread_start_us":88,"threads_started":1}
I20260620 04:30:31.021744 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.015505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.0.98.2:34031)
0620 04:30:31.015939 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:31.015949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:31.015972 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:30:31.016247 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:31.016253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:31.016464 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:31.016615 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:31.017118 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.018364 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.019979 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.020768 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.020872 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.021031 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:31.021509 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 04:30:31.021595 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":138,"threads_started":1}
I20260620 04:30:31.021720 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.016374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.0.98.1:43617)
0620 04:30:31.017047 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:31.017052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:31.017073 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:30:31.017141 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:31.017146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:31.017330 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:31.017479 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:31.017675 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.018736 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.020084 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.020811 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.020890 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.021031 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:31.021481 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 04:30:31.021575 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":96,"threads_started":1}
I20260620 04:30:31.022024 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.015359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34031 (local address 127.0.0.1:39832)
0620 04:30:31.015853 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:31.015903 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:31.016027 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:31.016809 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:31.016815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:31.016838 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:31.016968 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:31.016975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.018488 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.018492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:31.019841 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:31.019851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.021091 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.021095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:31.021216 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.021260 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 04:30:31.021330 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:30:31.021403 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":77,"thread_start_us":97,"threads_started":1}
I20260620 04:30:31.024572 4748 tablet_service.cc:1467] Processing CreateTablet for tablet bce3ffe2b208441180d1066a7094f8bd (DEFAULT_TABLE table=test-table [id=7de4aaf1498545ea82adb814e073dc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:31.024434 4881 tablet_service.cc:1467] Processing CreateTablet for tablet bce3ffe2b208441180d1066a7094f8bd (DEFAULT_TABLE table=test-table [id=7de4aaf1498545ea82adb814e073dc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:31.024431 4615 tablet_service.cc:1467] Processing CreateTablet for tablet bce3ffe2b208441180d1066a7094f8bd (DEFAULT_TABLE table=test-table [id=7de4aaf1498545ea82adb814e073dc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:31.025570 4615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce3ffe2b208441180d1066a7094f8bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:31.025571 4748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce3ffe2b208441180d1066a7094f8bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:31.025719 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce3ffe2b208441180d1066a7094f8bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:31.031615 4964 tablet_bootstrap.cc:492] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: Bootstrap starting.
I20260620 04:30:31.033113 4965 tablet_bootstrap.cc:492] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f: Bootstrap starting.
I20260620 04:30:31.033337 4966 tablet_bootstrap.cc:492] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: Bootstrap starting.
I20260620 04:30:31.033948 4964 tablet_bootstrap.cc:654] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:31.035254 4966 tablet_bootstrap.cc:654] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:31.035254 4965 tablet_bootstrap.cc:654] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:31.035351 4964 log.cc:824] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:31.036425 4966 log.cc:824] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:31.036424 4965 log.cc:824] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:31.037269 4964 tablet_bootstrap.cc:492] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: No bootstrap required, opened a new log
I20260620 04:30:31.037513 4964 ts_tablet_manager.cc:1397] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:30:31.038452 4966 tablet_bootstrap.cc:492] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: No bootstrap required, opened a new log
I20260620 04:30:31.038450 4965 tablet_bootstrap.cc:492] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f: No bootstrap required, opened a new log
I20260620 04:30:31.038635 4965 ts_tablet_manager.cc:1397] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:30:31.038743 4966 ts_tablet_manager.cc:1397] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:30:31.041023 4964 raft_consensus.cc:348] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } } peers { permanent_uuid: "6f75843c197744c2835829739f7673e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } } peers { permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 } }
I20260620 04:30:31.041299 4964 raft_consensus.cc:374] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:31.041378 4964 raft_consensus.cc:729] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64190df49e9e4636820ca97303f92a50, State: Initialized, Role: FOLLOWER
I20260620 04:30:31.041819 4964 consensus_queue.cc:260] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } } peers { permanent_uuid: "6f75843c197744c2835829739f7673e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } } peers { permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 } }
I20260620 04:30:31.042407 4966 raft_consensus.cc:348] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } } peers { permanent_uuid: "6f75843c197744c2835829739f7673e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } } peers { permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 } }
I20260620 04:30:31.042407 4965 raft_consensus.cc:348] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } } peers { permanent_uuid: "6f75843c197744c2835829739f7673e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } } peers { permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 } }
I20260620 04:30:31.042668 4965 raft_consensus.cc:374] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:31.042672 4966 raft_consensus.cc:374] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:31.042716 4964 ts_tablet_manager.cc:1428] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:30:31.042733 4966 raft_consensus.cc:729] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f75843c197744c2835829739f7673e8, State: Initialized, Role: FOLLOWER
I20260620 04:30:31.042797 4965 raft_consensus.cc:729] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbcbc1a1a7a04008a99f42ec8e39361f, State: Initialized, Role: FOLLOWER
I20260620 04:30:31.043066 4814 heartbeater.cc:503] Master 127.0.98.62:39595 was elected leader, sending a full tablet report...
I20260620 04:30:31.043164 4966 consensus_queue.cc:260] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } } peers { permanent_uuid: "6f75843c197744c2835829739f7673e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } } peers { permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 } }
I20260620 04:30:31.043270 4965 consensus_queue.cc:260] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } } peers { permanent_uuid: "6f75843c197744c2835829739f7673e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } } peers { permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 } }
I20260620 04:30:31.044097 4681 heartbeater.cc:503] Master 127.0.98.62:39595 was elected leader, sending a full tablet report...
I20260620 04:30:31.044121 4947 heartbeater.cc:503] Master 127.0.98.62:39595 was elected leader, sending a full tablet report...
I20260620 04:30:31.044220 4966 ts_tablet_manager.cc:1428] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:30:31.045608 4965 ts_tablet_manager.cc:1428] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 04:30:31.196694 4971 raft_consensus.cc:482] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:31.196918 4971 raft_consensus.cc:504] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } } peers { permanent_uuid: "6f75843c197744c2835829739f7673e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } } peers { permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 } }
I20260620 04:30:31.198096 4971 leader_election.cc:288] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64190df49e9e4636820ca97303f92a50 (127.0.98.2:34031), dbcbc1a1a7a04008a99f42ec8e39361f (127.0.98.1:43617)
I20260620 04:30:31.201642 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.198553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55065 (local address 127.0.98.2:34031)
0620 04:30:31.198676 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:31.198679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:31.198691 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:30:31.198739 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:31.198741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:31.198780 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:31.198861 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:31.199149 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.199640 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.200677 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.201114 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.201204 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.201266 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:31.201433 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 04:30:31.201514 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 04:30:31.201642 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.198345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34031 (local address 127.0.98.3:55065)
0620 04:30:31.198450 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:31.198465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:31.198559 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:31.198892 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:31.198895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:31.198913 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:31.199024 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:31.199030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.199745 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.199749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:31.200546 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:31.200555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.201224 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.201228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:31.201326 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.201343 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 04:30:31.201408 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 04:30:31.201497 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260620 04:30:31.201824 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.198345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43617 (local address 127.0.98.3:44545)
0620 04:30:31.198790 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:31.198800 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:31.198890 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:31.199113 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:31.199115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:31.199130 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:31.199265 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:31.199270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.199955 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.199958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:31.200755 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:31.200763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.201403 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.201407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:31.201533 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.201549 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 04:30:31.201614 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 04:30:31.201678 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":147,"thread_start_us":104,"threads_started":1}
I20260620 04:30:31.201817 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.198442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44545 (local address 127.0.98.1:43617)
0620 04:30:31.198568 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:31.198573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:31.198880 (+ 307us) server_negotiation.cc:407] Connection header received
0620 04:30:31.198986 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:31.198989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:31.199033 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:31.199091 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:31.199361 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.199855 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.200874 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.201303 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.201391 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.201448 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:31.201641 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 04:30:31.201715 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:30:31.202378 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce3ffe2b208441180d1066a7094f8bd" candidate_uuid: "6f75843c197744c2835829739f7673e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64190df49e9e4636820ca97303f92a50" is_pre_election: true
I20260620 04:30:31.202390 4635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce3ffe2b208441180d1066a7094f8bd" candidate_uuid: "6f75843c197744c2835829739f7673e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" is_pre_election: true
I20260620 04:30:31.202652 4635 raft_consensus.cc:2454] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f75843c197744c2835829739f7673e8 in term 0.
I20260620 04:30:31.202656 4768 raft_consensus.cc:2454] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f75843c197744c2835829739f7673e8 in term 0.
I20260620 04:30:31.203104 4835 leader_election.cc:302] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f75843c197744c2835829739f7673e8, dbcbc1a1a7a04008a99f42ec8e39361f; no voters:
I20260620 04:30:31.203384 4971 raft_consensus.cc:2781] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:31.203459 4971 raft_consensus.cc:482] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:31.203509 4971 raft_consensus.cc:3037] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:31.204535 4971 raft_consensus.cc:504] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } } peers { permanent_uuid: "6f75843c197744c2835829739f7673e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } } peers { permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 } }
I20260620 04:30:31.204888 4971 leader_election.cc:288] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [CANDIDATE]: Term 1 election: Requested vote from peers 64190df49e9e4636820ca97303f92a50 (127.0.98.2:34031), dbcbc1a1a7a04008a99f42ec8e39361f (127.0.98.1:43617)
I20260620 04:30:31.205269 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce3ffe2b208441180d1066a7094f8bd" candidate_uuid: "6f75843c197744c2835829739f7673e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64190df49e9e4636820ca97303f92a50"
I20260620 04:30:31.205333 4635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce3ffe2b208441180d1066a7094f8bd" candidate_uuid: "6f75843c197744c2835829739f7673e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f"
I20260620 04:30:31.205395 4768 raft_consensus.cc:3037] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:31.205457 4635 raft_consensus.cc:3037] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:31.206446 4768 raft_consensus.cc:2454] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f75843c197744c2835829739f7673e8 in term 1.
I20260620 04:30:31.206455 4635 raft_consensus.cc:2454] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f75843c197744c2835829739f7673e8 in term 1.
I20260620 04:30:31.206715 4835 leader_election.cc:302] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f75843c197744c2835829739f7673e8, dbcbc1a1a7a04008a99f42ec8e39361f; no voters:
I20260620 04:30:31.206909 4971 raft_consensus.cc:2781] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:31.207111 4971 raft_consensus.cc:686] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 1 LEADER]: Becoming Leader. State: Replica: 6f75843c197744c2835829739f7673e8, State: Running, Role: LEADER
I20260620 04:30:31.207417 4971 consensus_queue.cc:237] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } } peers { permanent_uuid: "6f75843c197744c2835829739f7673e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } } peers { permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 } }
I20260620 04:30:31.210494 4489 catalog_manager.cc:5495] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f75843c197744c2835829739f7673e8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6f75843c197744c2835829739f7673e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f75843c197744c2835829739f7673e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 } health_report { overall_health: UNKNOWN } } }
W20260620 04:30:31.221276 4948 tablet.cc:2369] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:30:31.251057 4815 tablet.cc:2369] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:30:31.260612 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.256519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.0.98.3:43919)
0620 04:30:31.256667 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:31.256671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:31.256688 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:30:31.256777 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:31.256780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:31.256845 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:31.256955 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:31.257251 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.258080 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.258987 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:31.259322 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:31.259419 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:31.259701 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:31.260225 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:31.260345 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:31.260376 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 04:30:31.260437 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:30:31.267004 4635 raft_consensus.cc:1261] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f [term 1 FOLLOWER]: Refusing update from remote peer 6f75843c197744c2835829739f7673e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:31.267004 4768 raft_consensus.cc:1261] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50 [term 1 FOLLOWER]: Refusing update from remote peer 6f75843c197744c2835829739f7673e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:31.267719 4974 consensus_queue.cc:1035] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64190df49e9e4636820ca97303f92a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:31.267912 4971 consensus_queue.cc:1035] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbcbc1a1a7a04008a99f42ec8e39361f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:31.276692 4979 mvcc.cc:205] Tried to move back new op lower bound from 7298784588861657088 to 7298784588627890176. Current Snapshot: MvccSnapshot[applied={T|T < 7298784588861657088}]
I20260620 04:30:31.277310 4978 mvcc.cc:205] Tried to move back new op lower bound from 7298784588861657088 to 7298784588627890176. Current Snapshot: MvccSnapshot[applied={T|T < 7298784588861657088}]
I20260620 04:30:31.283149 4489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:31.283310 4489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:31.285524 4489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e867f8a978614ecea8147a970934c0ff: Sending DeleteTablet for 3 replicas of tablet bce3ffe2b208441180d1066a7094f8bd
I20260620 04:30:31.286274 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet bce3ffe2b208441180d1066a7094f8bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56178
I20260620 04:30:31.286262 4748 tablet_service.cc:1514] Processing DeleteTablet for tablet bce3ffe2b208441180d1066a7094f8bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39832
I20260620 04:30:31.286804 4987 tablet_replica.cc:330] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: stopping tablet replica
I20260620 04:30:31.286819 4615 tablet_service.cc:1514] Processing DeleteTablet for tablet bce3ffe2b208441180d1066a7094f8bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56206
I20260620 04:30:31.287257 4987 raft_consensus.cc:2229] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:31.287287 4988 tablet_replica.cc:330] T bce3ffe2b208441180d1066a7094f8bd P dbcbc1a1a7a04008a99f42ec8e39361f: stopping tablet replica
I20260620 04:30:31.287621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 4553
I20260620 04:30:31.287652 4987 raft_consensus.cc:2258] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:31.289266 4986 tablet_replica.cc:330] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: stopping tablet replica
I20260620 04:30:31.289609 4986 raft_consensus.cc:2229] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:31.290011 4986 raft_consensus.cc:2258] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:31.290115 4987 ts_tablet_manager.cc:1905] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:31.291458 4986 ts_tablet_manager.cc:1905] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:31.292907 4987 ts_tablet_manager.cc:1918] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:31.293056 4987 log.cc:1196] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/bce3ffe2b208441180d1066a7094f8bd
I20260620 04:30:31.293385 4987 ts_tablet_manager.cc:1939] T bce3ffe2b208441180d1066a7094f8bd P 64190df49e9e4636820ca97303f92a50: Deleting consensus metadata
W20260620 04:30:31.294193 4476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43617 (error 108)
I20260620 04:30:31.294240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 4686
I20260620 04:30:31.295877 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.295576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43617 (local address 127.0.0.1:56208)
0620 04:30:31.295716 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:31.295787 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 04:30:31.296109 4476 catalog_manager.cc:4584] TS dbcbc1a1a7a04008a99f42ec8e39361f (127.0.98.1:43617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bce3ffe2b208441180d1066a7094f8bd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43617: connect: Connection refused (error 111)
I20260620 04:30:31.300002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 4819
I20260620 04:30:31.300187 4986 ts_tablet_manager.cc:1918] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:31.300283 4986 log.cc:1196] T bce3ffe2b208441180d1066a7094f8bd P 6f75843c197744c2835829739f7673e8: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/bce3ffe2b208441180d1066a7094f8bd
I20260620 04:30:31.301029 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.300131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34031 (local address 127.0.0.1:39846)
0620 04:30:31.300855 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:31.300909 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641}
I20260620 04:30:31.304747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 4460
2026-06-20T04:30:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2308 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T04:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:31Z Disabled control of system clock
I20260620 04:30:31.332858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40631
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 04:30:31.441059 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:31.441345 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:31.441385 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:31.445503 4993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:31.445572 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:31.445588 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:31.445603 4993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:31.445617 4993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:31.450405 4993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:31.451413 4993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:31.452730 4993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:31.460901 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:31.460915 4999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:31.460963 4993 server_base.cc:1034] running on GCE node
W20260620 04:30:31.460909 5001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:31.461784 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:31.462890 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:31.464061 4993 hybrid_clock.cc:648] HybridClock initialized: now 1781929831464040 us; error 31 us; skew 500 ppm
I20260620 04:30:31.466039 4993 webserver.cc:460] Webserver started at http://127.0.98.62:44985/ using document root <none> and password file <none>
I20260620 04:30:31.466609 4993 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:31.466676 4993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:31.466881 4993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:31.468988 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "7f92513e84ff463c8eca8ae3ed5d5770"
format_stamp: "Formatted at 2026-06-20 04:30:31 on dist-test-slave-rzqm"
server_key: "ceadc074bc83ce13b135ea3bcdab1b74"
server_key_iv: "952836ccb3197d3ce169aee42609f742"
server_key_version: "encryptionkey@0"
I20260620 04:30:31.469585 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f92513e84ff463c8eca8ae3ed5d5770"
format_stamp: "Formatted at 2026-06-20 04:30:31 on dist-test-slave-rzqm"
server_key: "ceadc074bc83ce13b135ea3bcdab1b74"
server_key_iv: "952836ccb3197d3ce169aee42609f742"
server_key_version: "encryptionkey@0"
I20260620 04:30:31.473417 4993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 04:30:31.475835 5007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:31.477241 4993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:31.477382 4993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "7f92513e84ff463c8eca8ae3ed5d5770"
format_stamp: "Formatted at 2026-06-20 04:30:31 on dist-test-slave-rzqm"
server_key: "ceadc074bc83ce13b135ea3bcdab1b74"
server_key_iv: "952836ccb3197d3ce169aee42609f742"
server_key_version: "encryptionkey@0"
I20260620 04:30:31.477504 4993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:31.482281 4993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:31.482909 4993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:31.483076 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:31.489761 4993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40631
I20260620 04:30:31.489785 5059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40631 every 8 connection(s)
I20260620 04:30:31.490733 4993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:31.494037 5060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:31.498068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 4993
I20260620 04:30:31.498180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:31.498438 392 external_mini_cluster.cc:1428] Setting key e487ea5e96a9e4399b1fc011e781315e
I20260620 04:30:31.499966 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770: Bootstrap starting.
I20260620 04:30:31.502295 5060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:31.503072 5060 log.cc:824] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:31.504061 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.499877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.0.98.62:40631)
0620 04:30:31.500529 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:31.500545 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:31.500581 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:30:31.501290 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:31.501308 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:31.501680 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:31.502010 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:31.502792 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:31.502797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:31.502808 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:31.502818 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:31.502872 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:31.502889 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:31.502980 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:31.503096 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 04:30:31.503366 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":133,"threads_started":1}
I20260620 04:30:31.504945 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770: No bootstrap required, opened a new log
I20260620 04:30:31.508554 5060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f92513e84ff463c8eca8ae3ed5d5770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40631 } }
I20260620 04:30:31.508792 5060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:31.508834 5060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f92513e84ff463c8eca8ae3ed5d5770, State: Initialized, Role: FOLLOWER
I20260620 04:30:31.509241 5060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f92513e84ff463c8eca8ae3ed5d5770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40631 } }
I20260620 04:30:31.509361 5060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:31.509407 5060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:31.509477 5060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:31.510341 5060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f92513e84ff463c8eca8ae3ed5d5770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40631 } }
I20260620 04:30:31.510655 5060 leader_election.cc:302] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f92513e84ff463c8eca8ae3ed5d5770; no voters:
I20260620 04:30:31.510928 5060 leader_election.cc:288] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:31.511034 5065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:31.511229 5065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [term 1 LEADER]: Becoming Leader. State: Replica: 7f92513e84ff463c8eca8ae3ed5d5770, State: Running, Role: LEADER
I20260620 04:30:31.511538 5065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f92513e84ff463c8eca8ae3ed5d5770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40631 } }
I20260620 04:30:31.512137 5060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:31.513216 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f92513e84ff463c8eca8ae3ed5d5770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f92513e84ff463c8eca8ae3ed5d5770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40631 } } }
I20260620 04:30:31.513339 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:31.513588 5067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f92513e84ff463c8eca8ae3ed5d5770. Latest consensus state: current_term: 1 leader_uuid: "7f92513e84ff463c8eca8ae3ed5d5770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f92513e84ff463c8eca8ae3ed5d5770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40631 } } }
I20260620 04:30:31.513671 5067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:31.513893 5074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:31.517024 5074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:31.523144 5074 catalog_manager.cc:1347] Generated new cluster ID: d6a2a49d28a440918da3cc439365be65
I20260620 04:30:31.523232 5074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:31.538197 5074 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:31.538579 5074 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:31.551927 5074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770: Generated new TSK 0
I20260620 04:30:31.552834 5074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:30:31.559074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40631
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 04:30:31.672842 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:31.673167 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:31.673213 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:31.677222 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:31.677284 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:31.677338 5084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:31.682296 5084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:31.683398 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:31.684780 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:31.694414 5089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:31.694480 5090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:31.694465 5092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:31.694746 5084 server_base.cc:1034] running on GCE node
I20260620 04:30:31.695179 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:31.695842 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:31.697048 5084 hybrid_clock.cc:648] HybridClock initialized: now 1781929831697016 us; error 40 us; skew 500 ppm
I20260620 04:30:31.698953 5084 webserver.cc:460] Webserver started at http://127.0.98.1:35643/ using document root <none> and password file <none>
I20260620 04:30:31.699540 5084 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:31.699608 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:31.699810 5084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:31.701922 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "3baaf2732cc94a5086c5ba77fb92b81e"
format_stamp: "Formatted at 2026-06-20 04:30:31 on dist-test-slave-rzqm"
server_key: "d3cede1a90892679c20e6fa56bd27369"
server_key_iv: "02b48ec3d09359d81d71e9330ec05c53"
server_key_version: "encryptionkey@0"
I20260620 04:30:31.702474 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3baaf2732cc94a5086c5ba77fb92b81e"
format_stamp: "Formatted at 2026-06-20 04:30:31 on dist-test-slave-rzqm"
server_key: "d3cede1a90892679c20e6fa56bd27369"
server_key_iv: "02b48ec3d09359d81d71e9330ec05c53"
server_key_version: "encryptionkey@0"
I20260620 04:30:31.706123 5084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 04:30:31.708524 5098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:31.709770 5084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:30:31.709877 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "3baaf2732cc94a5086c5ba77fb92b81e"
format_stamp: "Formatted at 2026-06-20 04:30:31 on dist-test-slave-rzqm"
server_key: "d3cede1a90892679c20e6fa56bd27369"
server_key_iv: "02b48ec3d09359d81d71e9330ec05c53"
server_key_version: "encryptionkey@0"
I20260620 04:30:31.709973 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:31.714542 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:31.715126 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:31.715284 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:31.715796 5084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:31.716666 5084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:31.716722 5084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:31.716773 5084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:31.716797 5084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:31.724888 5084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34557
I20260620 04:30:31.724929 5211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34557 every 8 connection(s)
I20260620 04:30:31.725922 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:31.732210 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.727811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59867 (local address 127.0.98.62:40631)
0620 04:30:31.727951 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:31.727955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:31.729109 (+ 1154us) server_negotiation.cc:407] Connection header received
0620 04:30:31.730408 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:31.730411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:31.730450 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:31.730523 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:31.731728 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:31.731732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:31.731734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:31.731746 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:31.731768 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:31.731776 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:31.731882 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:31.732011 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 04:30:31.732058 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:30:31.733100 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.728234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40631 (local address 127.0.98.1:59867)
0620 04:30:31.728929 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:31.728971 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:31.730171 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:31.730687 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:31.730694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:31.730796 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:31.731422 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:30:31.731424 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:31.731468 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:30:31.731545 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:31.731851 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:31.731856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:31.731861 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:30:31.731995 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 04:30:31.732208 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":110,"threads_started":1}
I20260620 04:30:31.734457 5212 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40631
I20260620 04:30:31.734735 5212 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:31.735229 5212 heartbeater.cc:511] Master 127.0.98.62:40631 requested a full tablet report, sending...
I20260620 04:30:31.735291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 5084
I20260620 04:30:31.735391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:31.735679 392 external_mini_cluster.cc:1428] Setting key f9e4f430baa30c53e824458f41f85943
I20260620 04:30:31.736760 5024 ts_manager.cc:194] Registered new tserver with Master: 3baaf2732cc94a5086c5ba77fb92b81e (127.0.98.1:34557)
I20260620 04:30:31.737563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40631
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 04:30:31.847384 5215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:31.847671 5215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:31.847710 5215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:31.851752 5215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:31.851841 5215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:31.851902 5215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:31.856943 5215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:31.858106 5215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:31.859472 5215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:31.868845 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:31.868870 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:31.869099 5215 server_base.cc:1034] running on GCE node
W20260620 04:30:31.869225 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:31.869632 5215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:31.870272 5215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:31.871433 5215 hybrid_clock.cc:648] HybridClock initialized: now 1781929831871411 us; error 33 us; skew 500 ppm
I20260620 04:30:31.873317 5215 webserver.cc:460] Webserver started at http://127.0.98.2:45813/ using document root <none> and password file <none>
I20260620 04:30:31.873870 5215 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:31.873934 5215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:31.874140 5215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:31.876195 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4"
format_stamp: "Formatted at 2026-06-20 04:30:31 on dist-test-slave-rzqm"
server_key: "47adfad7272ffb8fd40206aa372daa9c"
server_key_iv: "d7d0ba2e5d5e42d9a1ae5c4f6bc02efa"
server_key_version: "encryptionkey@0"
I20260620 04:30:31.876742 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4"
format_stamp: "Formatted at 2026-06-20 04:30:31 on dist-test-slave-rzqm"
server_key: "47adfad7272ffb8fd40206aa372daa9c"
server_key_iv: "d7d0ba2e5d5e42d9a1ae5c4f6bc02efa"
server_key_version: "encryptionkey@0"
I20260620 04:30:31.880483 5215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:30:31.883041 5229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:31.884362 5215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:31.884486 5215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4"
format_stamp: "Formatted at 2026-06-20 04:30:31 on dist-test-slave-rzqm"
server_key: "47adfad7272ffb8fd40206aa372daa9c"
server_key_iv: "d7d0ba2e5d5e42d9a1ae5c4f6bc02efa"
server_key_version: "encryptionkey@0"
I20260620 04:30:31.884596 5215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:31.889446 5215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:31.890080 5215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:31.890242 5215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:31.890910 5215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:31.891837 5215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:31.891883 5215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:31.891921 5215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:31.891942 5215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:31.900113 5215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36665
I20260620 04:30:31.900182 5342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36665 every 8 connection(s)
I20260620 04:30:31.901073 5215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:31.903407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 5215
I20260620 04:30:31.903517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:31.903771 392 external_mini_cluster.cc:1428] Setting key 6d87d0fd0d05d1a5fe282c801d0780b6
I20260620 04:30:31.905594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40631
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 04:30:31.908198 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.902883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38321 (local address 127.0.98.62:40631)
0620 04:30:31.903002 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:31.903005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:31.904216 (+ 1211us) server_negotiation.cc:407] Connection header received
0620 04:30:31.906014 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:31.906017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:31.906060 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:31.906130 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:31.907782 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:31.907785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:31.907787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:31.907802 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:31.907822 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:31.907830 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:31.907907 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:31.908055 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 04:30:31.908091 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:30:31.909448 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:31.903274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40631 (local address 127.0.98.2:38321)
0620 04:30:31.904044 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:31.904098 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:31.905747 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:31.906336 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:31.906359 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:31.906492 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:31.907414 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:30:31.907416 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:31.907479 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:30:31.907579 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:31.907910 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:31.907919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:31.907925 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:30:31.908099 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 04:30:31.908369 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":116,"threads_started":1}
I20260620 04:30:31.910733 5343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40631
I20260620 04:30:31.910969 5343 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:31.911376 5343 heartbeater.cc:511] Master 127.0.98.62:40631 requested a full tablet report, sending...
I20260620 04:30:31.912451 5024 ts_manager.cc:194] Registered new tserver with Master: 68acc4f7683d4636a9f1f9b0d5c1d3b4 (127.0.98.2:36665)
W20260620 04:30:32.018765 5346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:32.019045 5346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:32.019102 5346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:32.023257 5346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:32.023331 5346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:32.023413 5346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:32.028488 5346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:32.029701 5346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:32.031141 5346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:30:32.041127 5346 server_base.cc:1034] running on GCE node
W20260620 04:30:32.041107 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:32.041107 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:32.041100 5354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:32.041965 5346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:32.042626 5346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:32.043803 5346 hybrid_clock.cc:648] HybridClock initialized: now 1781929832043763 us; error 58 us; skew 500 ppm
I20260620 04:30:32.045847 5346 webserver.cc:460] Webserver started at http://127.0.98.3:37147/ using document root <none> and password file <none>
I20260620 04:30:32.046437 5346 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:32.046504 5346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:32.046695 5346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:32.048792 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "486674ac442b41aba1f307f776693419"
format_stamp: "Formatted at 2026-06-20 04:30:32 on dist-test-slave-rzqm"
server_key: "fadf4835b4d45057641927bc443084aa"
server_key_iv: "29e8e327933012427719ea2d96c27242"
server_key_version: "encryptionkey@0"
I20260620 04:30:32.049407 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "486674ac442b41aba1f307f776693419"
format_stamp: "Formatted at 2026-06-20 04:30:32 on dist-test-slave-rzqm"
server_key: "fadf4835b4d45057641927bc443084aa"
server_key_iv: "29e8e327933012427719ea2d96c27242"
server_key_version: "encryptionkey@0"
I20260620 04:30:32.053252 5346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:32.055886 5360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:32.057204 5346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:32.057345 5346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "486674ac442b41aba1f307f776693419"
format_stamp: "Formatted at 2026-06-20 04:30:32 on dist-test-slave-rzqm"
server_key: "fadf4835b4d45057641927bc443084aa"
server_key_iv: "29e8e327933012427719ea2d96c27242"
server_key_version: "encryptionkey@0"
I20260620 04:30:32.057451 5346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:32.062016 5346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:32.062603 5346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:32.062767 5346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:32.063284 5346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:32.064168 5346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:32.064213 5346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:32.064265 5346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:32.064291 5346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:32.072515 5346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43143
I20260620 04:30:32.072544 5473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43143 every 8 connection(s)
I20260620 04:30:32.073539 5346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:32.079470 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.075251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51629 (local address 127.0.98.62:40631)
0620 04:30:32.075375 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.075378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.076456 (+ 1078us) server_negotiation.cc:407] Connection header received
0620 04:30:32.077762 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.077765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.077805 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.077885 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:32.078905 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:32.078908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:32.078910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:32.078922 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:32.078943 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:32.078951 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:32.079028 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:32.079303 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 04:30:32.079360 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:30:32.080339 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.075644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40631 (local address 127.0.98.3:51629)
0620 04:30:32.076298 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:32.076341 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:32.077558 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:32.078045 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:32.078052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:32.078144 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:32.078645 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:30:32.078647 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:32.078688 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:30:32.078758 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:32.079070 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:32.079079 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:32.079084 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:30:32.079252 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 04:30:32.079503 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260620 04:30:32.081560 5474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40631
I20260620 04:30:32.081807 5474 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:32.082158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 5346
I20260620 04:30:32.082254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:32.082229 5474 heartbeater.cc:511] Master 127.0.98.62:40631 requested a full tablet report, sending...
I20260620 04:30:32.082510 392 external_mini_cluster.cc:1428] Setting key d0f5621f9efe7a7d4e330d966e1aae80
I20260620 04:30:32.083323 5024 ts_manager.cc:194] Registered new tserver with Master: 486674ac442b41aba1f307f776693419 (127.0.98.3:43143)
I20260620 04:30:32.084623 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:30:32.087809 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.086460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.0.98.62:40631)
0620 04:30:32.086572 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.086575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.086742 (+ 167us) server_negotiation.cc:407] Connection header received
0620 04:30:32.086849 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.086851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.086882 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.086946 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:32.087514 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:32.087517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:32.087519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:32.087528 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:32.087546 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:32.087553 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:32.087637 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:32.087672 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 04:30:32.087710 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:30:32.090816 5024 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:32.094043 5024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:32.106406 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.102809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.0.98.3:43143)
0620 04:30:32.103213 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.103224 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.103496 (+ 272us) server_negotiation.cc:407] Connection header received
0620 04:30:32.103559 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.103564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.103783 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.103936 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:32.105843 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:32.105849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:32.105862 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:32.105874 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:32.105918 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:32.105953 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:32.106064 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:32.106185 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 04:30:32.106257 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":189,"threads_started":1}
I20260620 04:30:32.106400 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.103405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36665 (local address 127.0.0.1:58960)
0620 04:30:32.103640 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:32.103652 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:32.103727 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:32.104760 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:32.104767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:32.104853 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:32.105774 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:30:32.105776 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:32.105785 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:30:32.105805 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:32.106089 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:32.106093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:32.106096 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 04:30:32.106147 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 04:30:32.106220 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
I20260620 04:30:32.106417 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.103735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34557 (local address 127.0.0.1:45714)
0620 04:30:32.103925 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:32.103937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:32.104004 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:32.105024 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:32.105028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:32.105043 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:32.105574 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:30:32.105580 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:32.105631 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:30:32.105743 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:32.106202 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:32.106205 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:32.106208 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 04:30:32.106254 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 04:30:32.106279 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260620 04:30:32.106418 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.102675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43143 (local address 127.0.0.1:53618)
0620 04:30:32.103070 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:32.103294 (+ 224us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:32.103407 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:32.105101 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:32.105104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:32.105119 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:32.105637 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:30:32.105639 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:32.105651 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:30:32.105722 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:32.106039 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:32.106048 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:32.106054 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:30:32.106173 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 04:30:32.106225 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":103,"threads_started":1}
I20260620 04:30:32.107401 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.103586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.0.98.2:36665)
0620 04:30:32.104333 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.104344 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.104369 (+ 25us) server_negotiation.cc:407] Connection header received
0620 04:30:32.104421 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.104426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.104594 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.104732 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:32.105910 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:32.105915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:32.105923 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:32.105934 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:32.105968 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:32.105994 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:32.106992 (+ 998us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:32.107047 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 04:30:32.107116 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":129,"threads_started":1}
I20260620 04:30:32.107924 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.104205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.0.98.1:34557)
0620 04:30:32.104607 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.104618 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.104641 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:30:32.104688 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.104692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.104856 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.105279 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:32.105976 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:32.105981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:32.105996 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:32.106019 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:32.106070 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:32.106099 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:32.107517 (+ 1418us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:32.107586 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 04:30:32.107702 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":119,"threads_started":1}
I20260620 04:30:32.109892 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 1574c599c3fa42e482ebe6f3c278957b (DEFAULT_TABLE table=test-table [id=5c3ebb728d8846e9bd9d27f498fed708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:32.109898 5277 tablet_service.cc:1467] Processing CreateTablet for tablet 1574c599c3fa42e482ebe6f3c278957b (DEFAULT_TABLE table=test-table [id=5c3ebb728d8846e9bd9d27f498fed708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:32.111037 5277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1574c599c3fa42e482ebe6f3c278957b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:32.111038 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1574c599c3fa42e482ebe6f3c278957b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:32.110812 5146 tablet_service.cc:1467] Processing CreateTablet for tablet 1574c599c3fa42e482ebe6f3c278957b (DEFAULT_TABLE table=test-table [id=5c3ebb728d8846e9bd9d27f498fed708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:32.111774 5146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1574c599c3fa42e482ebe6f3c278957b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:32.118256 5491 tablet_bootstrap.cc:492] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: Bootstrap starting.
I20260620 04:30:32.118980 5493 tablet_bootstrap.cc:492] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: Bootstrap starting.
I20260620 04:30:32.121058 5493 tablet_bootstrap.cc:654] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:32.121371 5492 tablet_bootstrap.cc:492] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e: Bootstrap starting.
I20260620 04:30:32.121390 5491 tablet_bootstrap.cc:654] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:32.121956 5493 log.cc:824] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:32.122375 5491 log.cc:824] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:32.123490 5492 tablet_bootstrap.cc:654] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:32.124228 5493 tablet_bootstrap.cc:492] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: No bootstrap required, opened a new log
I20260620 04:30:32.124312 5492 log.cc:824] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:32.124450 5493 ts_tablet_manager.cc:1397] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 04:30:32.124778 5491 tablet_bootstrap.cc:492] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: No bootstrap required, opened a new log
I20260620 04:30:32.125082 5491 ts_tablet_manager.cc:1397] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260620 04:30:32.126873 5492 tablet_bootstrap.cc:492] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e: No bootstrap required, opened a new log
I20260620 04:30:32.127067 5492 ts_tablet_manager.cc:1397] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:30:32.128494 5493 raft_consensus.cc:348] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 } } peers { permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34557 } }
I20260620 04:30:32.128736 5493 raft_consensus.cc:374] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:32.128787 5493 raft_consensus.cc:729] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68acc4f7683d4636a9f1f9b0d5c1d3b4, State: Initialized, Role: FOLLOWER
I20260620 04:30:32.129261 5493 consensus_queue.cc:260] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 } } peers { permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34557 } }
I20260620 04:30:32.129943 5343 heartbeater.cc:503] Master 127.0.98.62:40631 was elected leader, sending a full tablet report...
I20260620 04:30:32.130314 5493 ts_tablet_manager.cc:1428] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:30:32.131577 5492 raft_consensus.cc:348] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 } } peers { permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34557 } }
I20260620 04:30:32.131592 5491 raft_consensus.cc:348] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 } } peers { permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34557 } }
I20260620 04:30:32.131892 5492 raft_consensus.cc:374] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:32.131894 5491 raft_consensus.cc:374] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:32.131949 5492 raft_consensus.cc:729] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3baaf2732cc94a5086c5ba77fb92b81e, State: Initialized, Role: FOLLOWER
I20260620 04:30:32.131954 5491 raft_consensus.cc:729] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486674ac442b41aba1f307f776693419, State: Initialized, Role: FOLLOWER
I20260620 04:30:32.132431 5492 consensus_queue.cc:260] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 } } peers { permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34557 } }
I20260620 04:30:32.132537 5491 consensus_queue.cc:260] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 } } peers { permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34557 } }
I20260620 04:30:32.133620 5492 ts_tablet_manager.cc:1428] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:30:32.133894 5491 ts_tablet_manager.cc:1428] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: Time spent starting tablet: real 0.009s user 0.003s sys 0.005s
I20260620 04:30:32.134240 5212 heartbeater.cc:503] Master 127.0.98.62:40631 was elected leader, sending a full tablet report...
I20260620 04:30:32.134240 5474 heartbeater.cc:503] Master 127.0.98.62:40631 was elected leader, sending a full tablet report...
W20260620 04:30:32.152022 5344 tablet.cc:2369] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:30:32.166376 5498 raft_consensus.cc:482] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:32.166590 5498 raft_consensus.cc:504] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 } } peers { permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34557 } }
I20260620 04:30:32.167685 5498 leader_election.cc:288] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 486674ac442b41aba1f307f776693419 (127.0.98.3:43143), 68acc4f7683d4636a9f1f9b0d5c1d3b4 (127.0.98.2:36665)
I20260620 04:30:32.169561 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.168084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57087 (local address 127.0.98.3:43143)
0620 04:30:32.168206 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.168211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.168227 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:30:32.168368 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.168372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.168409 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.168475 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:32.169194 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:32.169198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:32.169201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:32.169214 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:32.169236 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:32.169246 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:32.169328 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:32.169397 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 04:30:32.169435 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:30:32.169561 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.167973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43143 (local address 127.0.98.1:57087)
0620 04:30:32.168107 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:32.168124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:32.168266 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:32.168511 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:32.168515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:32.168530 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:32.169025 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:30:32.169028 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:32.169040 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:30:32.169072 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:32.169322 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:32.169325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:32.169328 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 04:30:32.169400 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 04:30:32.169435 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 04:30:32.170150 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.168426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34915 (local address 127.0.98.2:36665)
0620 04:30:32.168541 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.168546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.168847 (+ 301us) server_negotiation.cc:407] Connection header received
0620 04:30:32.168932 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.168935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.168970 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.169061 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:32.169665 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:32.169669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:32.169672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:32.169687 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:32.169710 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:32.169722 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:32.169793 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:32.170019 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 04:30:32.170057 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:30:32.170184 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.168303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36665 (local address 127.0.98.1:34915)
0620 04:30:32.168756 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:32.168768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:32.168848 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:32.169072 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:32.169075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:32.169090 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:32.169506 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:30:32.169508 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:32.169518 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:30:32.169545 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:32.169927 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:32.169929 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:32.169932 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 04:30:32.170012 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:30:32.170046 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":135,"threads_started":1}
I20260620 04:30:32.170267 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1574c599c3fa42e482ebe6f3c278957b" candidate_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486674ac442b41aba1f307f776693419" is_pre_election: true
I20260620 04:30:32.170581 5428 raft_consensus.cc:2454] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3baaf2732cc94a5086c5ba77fb92b81e in term 0.
I20260620 04:30:32.170928 5297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1574c599c3fa42e482ebe6f3c278957b" candidate_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" is_pre_election: true
I20260620 04:30:32.171016 5100 leader_election.cc:302] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3baaf2732cc94a5086c5ba77fb92b81e, 486674ac442b41aba1f307f776693419; no voters:
I20260620 04:30:32.171173 5297 raft_consensus.cc:2454] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3baaf2732cc94a5086c5ba77fb92b81e in term 0.
I20260620 04:30:32.171275 5498 raft_consensus.cc:2781] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:32.171360 5498 raft_consensus.cc:482] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:32.171399 5498 raft_consensus.cc:3037] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:32.172399 5498 raft_consensus.cc:504] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 } } peers { permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34557 } }
I20260620 04:30:32.172713 5498 leader_election.cc:288] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [CANDIDATE]: Term 1 election: Requested vote from peers 486674ac442b41aba1f307f776693419 (127.0.98.3:43143), 68acc4f7683d4636a9f1f9b0d5c1d3b4 (127.0.98.2:36665)
I20260620 04:30:32.173091 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1574c599c3fa42e482ebe6f3c278957b" candidate_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486674ac442b41aba1f307f776693419"
I20260620 04:30:32.173229 5428 raft_consensus.cc:3037] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:32.173251 5297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1574c599c3fa42e482ebe6f3c278957b" candidate_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4"
I20260620 04:30:32.173369 5297 raft_consensus.cc:3037] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:32.174237 5428 raft_consensus.cc:2454] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3baaf2732cc94a5086c5ba77fb92b81e in term 1.
I20260620 04:30:32.174414 5297 raft_consensus.cc:2454] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3baaf2732cc94a5086c5ba77fb92b81e in term 1.
I20260620 04:30:32.174492 5100 leader_election.cc:302] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3baaf2732cc94a5086c5ba77fb92b81e, 486674ac442b41aba1f307f776693419; no voters:
I20260620 04:30:32.174657 5498 raft_consensus.cc:2781] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:32.174883 5498 raft_consensus.cc:686] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [term 1 LEADER]: Becoming Leader. State: Replica: 3baaf2732cc94a5086c5ba77fb92b81e, State: Running, Role: LEADER
I20260620 04:30:32.175205 5498 consensus_queue.cc:237] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 } } peers { permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34557 } }
I20260620 04:30:32.178759 5024 catalog_manager.cc:5495] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3baaf2732cc94a5086c5ba77fb92b81e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3baaf2732cc94a5086c5ba77fb92b81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34557 } health_report { overall_health: HEALTHY } } }
I20260620 04:30:32.187423 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.186005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.0.98.1:34557)
0620 04:30:32.186123 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.186127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.186141 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:32.186265 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.186267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.186360 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.186436 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:32.187092 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:32.187095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:32.187097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:32.187108 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:32.187128 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:32.187136 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:32.187211 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:32.187254 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 04:30:32.187287 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:30:32.193212 5428 raft_consensus.cc:1261] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419 [term 1 FOLLOWER]: Refusing update from remote peer 3baaf2732cc94a5086c5ba77fb92b81e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:32.193212 5297 raft_consensus.cc:1261] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4 [term 1 FOLLOWER]: Refusing update from remote peer 3baaf2732cc94a5086c5ba77fb92b81e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:32.193897 5501 consensus_queue.cc:1035] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [LEADER]: Connected to new peer: Peer: permanent_uuid: "486674ac442b41aba1f307f776693419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:32.194080 5498 consensus_queue.cc:1035] T 1574c599c3fa42e482ebe6f3c278957b P 3baaf2732cc94a5086c5ba77fb92b81e [LEADER]: Connected to new peer: Peer: permanent_uuid: "68acc4f7683d4636a9f1f9b0d5c1d3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:32.202193 5505 mvcc.cc:205] Tried to move back new op lower bound from 7298784592656375808 to 7298784592592080896. Current Snapshot: MvccSnapshot[applied={T|T < 7298784592656375808}]
I20260620 04:30:32.202627 5504 mvcc.cc:205] Tried to move back new op lower bound from 7298784592656375808 to 7298784592592080896. Current Snapshot: MvccSnapshot[applied={T|T < 7298784592656375808}]
I20260620 04:30:32.210412 5024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:32.210574 5024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:32.212785 5024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f92513e84ff463c8eca8ae3ed5d5770: Sending DeleteTablet for 3 replicas of tablet 1574c599c3fa42e482ebe6f3c278957b
I20260620 04:30:32.213635 5277 tablet_service.cc:1514] Processing DeleteTablet for tablet 1574c599c3fa42e482ebe6f3c278957b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:32 UTC) from {username='slave'} at 127.0.0.1:58960
I20260620 04:30:32.213653 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet 1574c599c3fa42e482ebe6f3c278957b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:32 UTC) from {username='slave'} at 127.0.0.1:45714
I20260620 04:30:32.214129 5515 tablet_replica.cc:330] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: stopping tablet replica
I20260620 04:30:32.214124 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 1574c599c3fa42e482ebe6f3c278957b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:32 UTC) from {username='slave'} at 127.0.0.1:53618
I20260620 04:30:32.214465 5515 raft_consensus.cc:2229] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:32.214636 5516 tablet_replica.cc:330] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: stopping tablet replica
I20260620 04:30:32.214808 5515 raft_consensus.cc:2258] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:32.215022 5516 raft_consensus.cc:2229] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:32.215209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 5084
I20260620 04:30:32.216493 5516 raft_consensus.cc:2258] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:32.216715 5515 ts_tablet_manager.cc:1905] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:32.217998 5516 ts_tablet_manager.cc:1905] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:32.219058 5515 ts_tablet_manager.cc:1918] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:32.219141 5515 log.cc:1196] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/1574c599c3fa42e482ebe6f3c278957b
I20260620 04:30:32.219399 5515 ts_tablet_manager.cc:1939] T 1574c599c3fa42e482ebe6f3c278957b P 68acc4f7683d4636a9f1f9b0d5c1d3b4: Deleting consensus metadata
I20260620 04:30:32.220466 5009 catalog_manager.cc:4841] TS 68acc4f7683d4636a9f1f9b0d5c1d3b4 (127.0.98.2:36665): tablet 1574c599c3fa42e482ebe6f3c278957b (table test-table [id=5c3ebb728d8846e9bd9d27f498fed708]) successfully deleted
I20260620 04:30:32.220530 5516 ts_tablet_manager.cc:1918] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:32.220607 5516 log.cc:1196] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/1574c599c3fa42e482ebe6f3c278957b
I20260620 04:30:32.220898 5516 ts_tablet_manager.cc:1939] T 1574c599c3fa42e482ebe6f3c278957b P 486674ac442b41aba1f307f776693419: Deleting consensus metadata
I20260620 04:30:32.222044 5009 catalog_manager.cc:4841] TS 486674ac442b41aba1f307f776693419 (127.0.98.3:43143): tablet 1574c599c3fa42e482ebe6f3c278957b (table test-table [id=5c3ebb728d8846e9bd9d27f498fed708]) successfully deleted
W20260620 04:30:32.222713 5009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34557 (error 108)
I20260620 04:30:32.222903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 5215
I20260620 04:30:32.225754 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.225489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34557 (local address 127.0.0.1:45740)
0620 04:30:32.225611 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:32.225681 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 04:30:32.225971 5009 catalog_manager.cc:4584] TS 3baaf2732cc94a5086c5ba77fb92b81e (127.0.98.1:34557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1574c599c3fa42e482ebe6f3c278957b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34557: connect: Connection refused (error 111)
I20260620 04:30:32.227936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 5346
I20260620 04:30:32.232492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 4993
2026-06-20T04:30:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (926 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 04:30:32.252468 392 mini_oidc.cc:152] Starting JWKS server
I20260620 04:30:32.254093 392 webserver.cc:460] Webserver started at https://127.0.0.1:34869/ using document root <none> and password file <none>
I20260620 04:30:32.254268 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 04:30:32.254521 392 webserver.cc:460] Webserver started at http://127.0.0.1:42863/ using document root <none> and password file <none>
2026-06-20T04:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:32Z Disabled control of system clock
I20260620 04:30:32.263260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37891
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37891
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34869/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/testchainca.cert with env {}
W20260620 04:30:32.372673 5523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:32.372961 5523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:32.373026 5523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:32.377094 5523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:32.377166 5523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:32.377179 5523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:32.377195 5523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:32.377209 5523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:30:32.377224 5523 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:30:32.377236 5523 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34869/jwks.json
I20260620 04:30:32.382736 5523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34869/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:32.383875 5523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:32.384927 5523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:30:32.393541 5523 server_base.cc:1034] running on GCE node
W20260620 04:30:32.393486 5528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:32.393519 5529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:32.393481 5531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:32.394476 5523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:32.395558 5523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:32.396737 5523 hybrid_clock.cc:648] HybridClock initialized: now 1781929832396717 us; error 32 us; skew 500 ppm
I20260620 04:30:32.398628 5523 webserver.cc:460] Webserver started at http://127.0.98.62:44881/ using document root <none> and password file <none>
I20260620 04:30:32.399195 5523 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:32.399256 5523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:32.399446 5523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:32.401562 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "610f69377f5f475fbfb57e5853c47ad9"
format_stamp: "Formatted at 2026-06-20 04:30:32 on dist-test-slave-rzqm"
server_key: "23242b8f6402b0d3868a7bca647ac23e"
server_key_iv: "7fdb6c0e06fc66f9ae93d9dd8854b58c"
server_key_version: "encryptionkey@0"
I20260620 04:30:32.402134 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "610f69377f5f475fbfb57e5853c47ad9"
format_stamp: "Formatted at 2026-06-20 04:30:32 on dist-test-slave-rzqm"
server_key: "23242b8f6402b0d3868a7bca647ac23e"
server_key_iv: "7fdb6c0e06fc66f9ae93d9dd8854b58c"
server_key_version: "encryptionkey@0"
I20260620 04:30:32.405992 5523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:30:32.408450 5537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:32.409857 5523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:32.409973 5523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "610f69377f5f475fbfb57e5853c47ad9"
format_stamp: "Formatted at 2026-06-20 04:30:32 on dist-test-slave-rzqm"
server_key: "23242b8f6402b0d3868a7bca647ac23e"
server_key_iv: "7fdb6c0e06fc66f9ae93d9dd8854b58c"
server_key_version: "encryptionkey@0"
I20260620 04:30:32.410074 5523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:32.431538 5523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:32.432265 5523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:32.432458 5523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:32.439308 5523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37891
I20260620 04:30:32.439301 5589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37891 every 8 connection(s)
I20260620 04:30:32.440277 5523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:32.443822 5590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:32.449234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 5523
I20260620 04:30:32.449384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:32.449661 392 external_mini_cluster.cc:1428] Setting key 090e01a54e289af9aca051e04e50e814
I20260620 04:30:32.450160 5590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9: Bootstrap starting.
I20260620 04:30:32.452725 5590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:32.453658 5590 log.cc:824] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:32.455953 5590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9: No bootstrap required, opened a new log
I20260620 04:30:32.459333 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.451126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.0.98.62:37891)
0620 04:30:32.451759 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.451774 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.451813 (+ 39us) server_negotiation.cc:407] Connection header received
0620 04:30:32.452487 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.452506 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.452984 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.453372 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:32.453760 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.454768 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.455974 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.456518 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.456647 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:32.457504 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:32.457509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:32.457523 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:32.457931 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:32.457990 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:32.458010 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:32.458087 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:32.458248 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 04:30:32.458558 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260620 04:30:32.460069 5590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610f69377f5f475fbfb57e5853c47ad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260620 04:30:32.460333 5590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:32.460394 5590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610f69377f5f475fbfb57e5853c47ad9, State: Initialized, Role: FOLLOWER
I20260620 04:30:32.460855 5590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610f69377f5f475fbfb57e5853c47ad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260620 04:30:32.461020 5590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:32.461091 5590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:32.461184 5590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:32.462227 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:30:32.462193 5590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610f69377f5f475fbfb57e5853c47ad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260620 04:30:32.462572 5590 leader_election.cc:302] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 610f69377f5f475fbfb57e5853c47ad9; no voters:
I20260620 04:30:32.462931 5590 leader_election.cc:288] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:32.463076 5595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:32.463280 5595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [term 1 LEADER]: Becoming Leader. State: Replica: 610f69377f5f475fbfb57e5853c47ad9, State: Running, Role: LEADER
I20260620 04:30:32.463613 5595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610f69377f5f475fbfb57e5853c47ad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } }
I20260620 04:30:32.464258 5590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:32.464875 5597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 610f69377f5f475fbfb57e5853c47ad9. Latest consensus state: current_term: 1 leader_uuid: "610f69377f5f475fbfb57e5853c47ad9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610f69377f5f475fbfb57e5853c47ad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } } }
I20260620 04:30:32.465024 5597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:32.465328 5596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "610f69377f5f475fbfb57e5853c47ad9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610f69377f5f475fbfb57e5853c47ad9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37891 } } }
I20260620 04:30:32.465425 5596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9 [sys.catalog]: This master's current role is: LEADER
W20260620 04:30:32.468865 5610 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:30:32.468940 5610 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:30:32.469029 5602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:32.469731 5602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:32.475428 5602 catalog_manager.cc:1347] Generated new cluster ID: a1bbcb971fbb4f7295ca253ae7851c6d
I20260620 04:30:32.475518 5602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:32.487560 5602 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:32.488510 5602 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:32.496096 5602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 610f69377f5f475fbfb57e5853c47ad9: Generated new TSK 0
I20260620 04:30:32.496846 5602 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 04:30:32.533746 5620 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 04:30:32.533872 5620 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 04:30:32.534363 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.509909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.0.98.62:37891)
0620 04:30:32.510039 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.510044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.510178 (+ 134us) server_negotiation.cc:407] Connection header received
0620 04:30:32.510405 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.510408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.510463 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.510556 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:30:32.510807 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.511357 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.512208 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.512434 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.512518 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:32.512847 (+ 329us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:30:32.533897 (+ 21050us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 04:30:32.534203 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 04:30:32.534477 5593 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 04:30:32.535630 5619 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:32.509767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37891 (local address 127.0.0.1:37002)
0620 04:30:32.510096 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:32.510108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:32.510238 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:32.510584 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:32.510587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:32.510595 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:30:32.510705 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:32.510710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.511467 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.511470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:32.512099 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:32.512107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.512519 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.512540 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:32.512629 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:32.512672 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:30:32.534156 (+ 21484us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 04:30:32.534432 (+ 276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:37891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 04:30:32.536644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 5523
2026-06-20T04:30:32Z chronyd exiting
I20260620 04:30:32.573086 392 test_util.cc:175] -----------------------------------------------
I20260620 04:30:32.573175 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781929807654528-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (322 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 04:30:32.574204 392 mini_oidc.cc:152] Starting JWKS server
I20260620 04:30:32.575048 392 webserver.cc:460] Webserver started at https://127.0.0.1:33515/ using document root <none> and password file <none>
I20260620 04:30:32.575141 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 04:30:32.575390 392 webserver.cc:460] Webserver started at http://127.0.0.1:37397/ using document root <none> and password file <none>
2026-06-20T04:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:32Z Disabled control of system clock
I20260620 04:30:32.583325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36661
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36661
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33515/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/testchainca.cert with env {}
W20260620 04:30:32.692988 5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:32.693331 5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:32.693399 5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:32.697585 5626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:32.697657 5626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:32.697678 5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:32.697700 5626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:32.697721 5626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:30:32.697747 5626 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:30:32.697773 5626 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33515/jwks.json
I20260620 04:30:32.703351 5626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33515/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:32.704478 5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:32.705536 5626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:32.714100 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:32.714100 5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:32.714110 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:32.714443 5626 server_base.cc:1034] running on GCE node
I20260620 04:30:32.715041 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:32.716302 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:32.717475 5626 hybrid_clock.cc:648] HybridClock initialized: now 1781929832717455 us; error 34 us; skew 500 ppm
I20260620 04:30:32.719370 5626 webserver.cc:460] Webserver started at http://127.0.98.62:46143/ using document root <none> and password file <none>
I20260620 04:30:32.719990 5626 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:32.720059 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:32.720266 5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:32.722445 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "07ccfedc18f2434292eff6e199808a0a"
format_stamp: "Formatted at 2026-06-20 04:30:32 on dist-test-slave-rzqm"
server_key: "e69dbd5f4d614c420bf7971b0e22634a"
server_key_iv: "4e18ad1b938b25693ee0fff0486a8db1"
server_key_version: "encryptionkey@0"
I20260620 04:30:32.722988 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "07ccfedc18f2434292eff6e199808a0a"
format_stamp: "Formatted at 2026-06-20 04:30:32 on dist-test-slave-rzqm"
server_key: "e69dbd5f4d614c420bf7971b0e22634a"
server_key_iv: "4e18ad1b938b25693ee0fff0486a8db1"
server_key_version: "encryptionkey@0"
I20260620 04:30:32.726725 5626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:30:32.729184 5640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:32.730518 5626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:32.730647 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "07ccfedc18f2434292eff6e199808a0a"
format_stamp: "Formatted at 2026-06-20 04:30:32 on dist-test-slave-rzqm"
server_key: "e69dbd5f4d614c420bf7971b0e22634a"
server_key_iv: "4e18ad1b938b25693ee0fff0486a8db1"
server_key_version: "encryptionkey@0"
I20260620 04:30:32.730763 5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:32.745424 5626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:32.746114 5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:32.746280 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:32.752717 5626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36661
I20260620 04:30:32.752746 5692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36661 every 8 connection(s)
I20260620 04:30:32.753691 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:32.756772 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:32.758762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 5626
I20260620 04:30:32.758883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:32.759183 392 external_mini_cluster.cc:1428] Setting key ccb79775674b666821ddbd3124084960
I20260620 04:30:32.763052 5693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a: Bootstrap starting.
I20260620 04:30:32.765591 5693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:32.766426 5693 log.cc:824] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:32.768333 5693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a: No bootstrap required, opened a new log
I20260620 04:30:32.768601 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.760581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.0.98.62:36661)
0620 04:30:32.761206 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.761222 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.761260 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:30:32.762154 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.762174 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.762642 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.762930 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:32.763306 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.764272 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.765463 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.765961 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.766075 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:32.766913 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:32.766919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:32.766932 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:32.767311 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:32.767366 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:32.767385 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:32.767497 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:32.767627 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 04:30:32.767874 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260620 04:30:32.771617 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:30:32.772305 5693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ccfedc18f2434292eff6e199808a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } }
I20260620 04:30:32.772599 5693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:32.772663 5693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ccfedc18f2434292eff6e199808a0a, State: Initialized, Role: FOLLOWER
I20260620 04:30:32.773159 5693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ccfedc18f2434292eff6e199808a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } }
I20260620 04:30:32.773324 5693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:32.773394 5693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:32.773487 5693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:32.774359 5693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ccfedc18f2434292eff6e199808a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } }
I20260620 04:30:32.774724 5693 leader_election.cc:302] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07ccfedc18f2434292eff6e199808a0a; no voters:
I20260620 04:30:32.775089 5693 leader_election.cc:288] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:32.775123 5698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:32.775331 5698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [term 1 LEADER]: Becoming Leader. State: Replica: 07ccfedc18f2434292eff6e199808a0a, State: Running, Role: LEADER
I20260620 04:30:32.775635 5698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ccfedc18f2434292eff6e199808a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } }
I20260620 04:30:32.776307 5693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:32.777254 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07ccfedc18f2434292eff6e199808a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ccfedc18f2434292eff6e199808a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } } }
I20260620 04:30:32.777387 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:32.777635 5699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07ccfedc18f2434292eff6e199808a0a. Latest consensus state: current_term: 1 leader_uuid: "07ccfedc18f2434292eff6e199808a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ccfedc18f2434292eff6e199808a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36661 } } }
I20260620 04:30:32.777737 5699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:32.777829 5707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:32.780851 5707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:32.787112 5707 catalog_manager.cc:1347] Generated new cluster ID: e6b270dbeb024054bfe124d3fcd01fe8
I20260620 04:30:32.787206 5707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:32.803272 5707 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:32.804656 5707 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:32.812278 5707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07ccfedc18f2434292eff6e199808a0a: Generated new TSK 0
I20260620 04:30:32.813234 5707 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 04:30:32.847558 5723 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 04:30:32.847649 5723 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 04:30:32.848470 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:32.831721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.0.98.62:36661)
0620 04:30:32.831869 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:32.831874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:32.831981 (+ 107us) server_negotiation.cc:407] Connection header received
0620 04:30:32.832138 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:32.832141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:32.832199 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:32.832321 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:30:32.832506 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.833389 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.834261 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.834634 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.834729 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:32.835045 (+ 316us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:30:32.848036 (+ 12991us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 04:30:32.848322 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 04:30:32.848671 5722 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:32.831621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36661 (local address 127.0.0.1:41504)
0620 04:30:32.831886 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:32.831899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:32.832006 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:32.832310 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:32.832312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:32.832317 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:30:32.832416 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:32.832421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.833496 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.833499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:32.834162 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:32.834171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:32.834737 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:32.834757 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:32.834825 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:32.834849 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:30:32.848294 (+ 13445us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 04:30:32.848486 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
W20260620 04:30:32.848543 5696 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 04:30:32.849247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 5626
2026-06-20T04:30:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (313 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 04:30:32.887848 392 mini_oidc.cc:152] Starting JWKS server
I20260620 04:30:32.888710 392 webserver.cc:460] Webserver started at https://127.0.0.1:45197/ using document root <none> and password file <none>
I20260620 04:30:32.888803 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 04:30:32.889097 392 webserver.cc:460] Webserver started at http://127.0.0.1:46753/ using document root <none> and password file <none>
2026-06-20T04:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:32Z Disabled control of system clock
I20260620 04:30:32.897395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39201
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39201
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45197/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 04:30:33.008888 5729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:33.009239 5729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:33.009282 5729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:33.013324 5729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:33.013394 5729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:33.013408 5729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:33.013424 5729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:33.013438 5729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:30:33.013453 5729 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:30:33.013465 5729 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45197/jwks.json
I20260620 04:30:33.018457 5729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45197/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:33.019627 5729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:33.021073 5729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:33.029523 5735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:33.029655 5729 server_base.cc:1034] running on GCE node
W20260620 04:30:33.029529 5734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:33.029538 5737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:33.030428 5729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:33.031529 5729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:33.032691 5729 hybrid_clock.cc:648] HybridClock initialized: now 1781929833032659 us; error 44 us; skew 500 ppm
I20260620 04:30:33.041105 5729 webserver.cc:460] Webserver started at http://127.0.98.62:33405/ using document root <none> and password file <none>
I20260620 04:30:33.041764 5729 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:33.041836 5729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:33.042042 5729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:33.044186 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1"
format_stamp: "Formatted at 2026-06-20 04:30:33 on dist-test-slave-rzqm"
server_key: "0e022870f805ddd465a6e8a238ad6191"
server_key_iv: "2920b6947a5ede599e99c90e6da9c88d"
server_key_version: "encryptionkey@0"
I20260620 04:30:33.044754 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1"
format_stamp: "Formatted at 2026-06-20 04:30:33 on dist-test-slave-rzqm"
server_key: "0e022870f805ddd465a6e8a238ad6191"
server_key_iv: "2920b6947a5ede599e99c90e6da9c88d"
server_key_version: "encryptionkey@0"
I20260620 04:30:33.048491 5729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260620 04:30:33.051034 5743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:33.052338 5729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:33.052493 5729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1"
format_stamp: "Formatted at 2026-06-20 04:30:33 on dist-test-slave-rzqm"
server_key: "0e022870f805ddd465a6e8a238ad6191"
server_key_iv: "2920b6947a5ede599e99c90e6da9c88d"
server_key_version: "encryptionkey@0"
I20260620 04:30:33.052604 5729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:33.067092 5729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:33.067797 5729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:33.067968 5729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:33.074692 5729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39201
I20260620 04:30:33.074715 5795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39201 every 8 connection(s)
I20260620 04:30:33.075620 5729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:33.078895 5796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:33.083149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 5729
I20260620 04:30:33.083271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:33.083547 392 external_mini_cluster.cc:1428] Setting key 2428025ad22ff7fe4f8cc28812874bbb
I20260620 04:30:33.085059 5796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1: Bootstrap starting.
I20260620 04:30:33.087306 5796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:33.088068 5796 log.cc:824] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:33.089957 5796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1: No bootstrap required, opened a new log
I20260620 04:30:33.092689 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:33.084968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.0.98.62:39201)
0620 04:30:33.085739 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:33.085755 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:33.085792 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:30:33.086419 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:33.086435 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:33.086826 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:33.087151 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:33.087579 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.088604 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.089545 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.090031 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.090163 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:33.090978 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:33.090983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:33.090995 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:33.091367 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:33.091445 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:33.091465 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:33.091537 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:33.091687 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 04:30:33.091947 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":148,"threads_started":1}
I20260620 04:30:33.093752 5796 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } }
I20260620 04:30:33.094034 5796 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:33.094089 5796 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d1f6e29e124e0c9af978fb42e2a9b1, State: Initialized, Role: FOLLOWER
I20260620 04:30:33.094516 5796 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } }
I20260620 04:30:33.094655 5796 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:33.094712 5796 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:33.094805 5796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:33.095695 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:30:33.095880 5796 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } }
I20260620 04:30:33.096235 5796 leader_election.cc:302] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d1f6e29e124e0c9af978fb42e2a9b1; no voters:
I20260620 04:30:33.096596 5796 leader_election.cc:288] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:33.096700 5801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:33.096944 5801 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [term 1 LEADER]: Becoming Leader. State: Replica: c9d1f6e29e124e0c9af978fb42e2a9b1, State: Running, Role: LEADER
I20260620 04:30:33.097294 5801 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } }
I20260620 04:30:33.097891 5796 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:33.098450 5802 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } } }
I20260620 04:30:33.098590 5802 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:33.098460 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d1f6e29e124e0c9af978fb42e2a9b1. Latest consensus state: current_term: 1 leader_uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d1f6e29e124e0c9af978fb42e2a9b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39201 } } }
I20260620 04:30:33.098807 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:33.098927 5808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:33.101981 5808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:33.108273 5808 catalog_manager.cc:1347] Generated new cluster ID: 8ba2a42c2c504765bd88b8e20601a3d8
I20260620 04:30:33.108366 5808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:33.116685 5808 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:33.117712 5808 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:33.125275 5808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9d1f6e29e124e0c9af978fb42e2a9b1: Generated new TSK 0
I20260620 04:30:33.126019 5808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:30:33.144877 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:33.139377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.0.98.62:39201)
0620 04:30:33.139519 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:33.139523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:33.139628 (+ 105us) server_negotiation.cc:407] Connection header received
0620 04:30:33.139793 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:33.139797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:33.139861 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:33.140003 (+ 142us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:30:33.140197 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.141086 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.142052 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.142416 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.142523 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:33.142810 (+ 287us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:30:33.144459 (+ 1649us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45197/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 04:30:33.144714 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45197/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 04:30:33.144955 5799 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45197/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 04:30:33.146495 5825 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:33.139262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39201 (local address 127.0.0.1:44460)
0620 04:30:33.139540 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:33.139554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:33.139656 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:33.140001 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:33.140003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:33.140009 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:30:33.140104 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:33.140109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.141217 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.141221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:33.141925 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:33.141935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.142520 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.142542 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:33.142608 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:33.142631 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:30:33.144644 (+ 2013us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45197/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 04:30:33.146311 (+ 1667us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39201: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45197/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":1399,"thread_start_us":89,"threads_started":1}
I20260620 04:30:33.146870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 5729
E20260620 04:30:33.156119 5826 webserver.cc:533] Webserver: sslize failed:
E20260620 04:30:33.156196 5826 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T04:30:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T04:30:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:33Z Disabled control of system clock
I20260620 04:30:33.193784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33985
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33985
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/testchainca.cert with env {}
W20260620 04:30:33.305549 5830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:33.305845 5830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:33.305886 5830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:33.309979 5830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:33.310048 5830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:33.310062 5830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:33.310078 5830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:33.310093 5830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 04:30:33.310108 5830 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 04:30:33.310120 5830 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 04:30:33.315771 5830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:33.316913 5830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:33.318087 5830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:33.326776 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:33.326815 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:33.326834 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:33.327117 5830 server_base.cc:1034] running on GCE node
I20260620 04:30:33.327623 5830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:33.328691 5830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:33.329890 5830 hybrid_clock.cc:648] HybridClock initialized: now 1781929833329851 us; error 53 us; skew 500 ppm
I20260620 04:30:33.331785 5830 webserver.cc:460] Webserver started at http://127.0.98.62:41147/ using document root <none> and password file <none>
I20260620 04:30:33.332348 5830 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:33.332409 5830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:33.332583 5830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:33.334688 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "ed9bca740ac3403fbc4473ec0c0122ca"
format_stamp: "Formatted at 2026-06-20 04:30:33 on dist-test-slave-rzqm"
server_key: "b3558471af3289d10f57c400eb2ab28e"
server_key_iv: "d5bfcbd8febfd793dacede875c24a8a7"
server_key_version: "encryptionkey@0"
I20260620 04:30:33.335214 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed9bca740ac3403fbc4473ec0c0122ca"
format_stamp: "Formatted at 2026-06-20 04:30:33 on dist-test-slave-rzqm"
server_key: "b3558471af3289d10f57c400eb2ab28e"
server_key_iv: "d5bfcbd8febfd793dacede875c24a8a7"
server_key_version: "encryptionkey@0"
I20260620 04:30:33.339028 5830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:30:33.341547 5844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:33.342916 5830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:33.343040 5830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "ed9bca740ac3403fbc4473ec0c0122ca"
format_stamp: "Formatted at 2026-06-20 04:30:33 on dist-test-slave-rzqm"
server_key: "b3558471af3289d10f57c400eb2ab28e"
server_key_iv: "d5bfcbd8febfd793dacede875c24a8a7"
server_key_version: "encryptionkey@0"
I20260620 04:30:33.343159 5830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:33.355522 5830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:33.356248 5830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:33.356434 5830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:33.363178 5830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33985
I20260620 04:30:33.363243 5896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33985 every 8 connection(s)
I20260620 04:30:33.364122 5830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:33.367334 5897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:33.369582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 5830
I20260620 04:30:33.369699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:33.370002 392 external_mini_cluster.cc:1428] Setting key 997fae5b8518a3fb257dee2ac10098a4
I20260620 04:30:33.374037 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca: Bootstrap starting.
I20260620 04:30:33.376682 5897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:33.377620 5897 log.cc:824] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:33.379984 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca: No bootstrap required, opened a new log
I20260620 04:30:33.381919 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:33.371584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59724 (local address 127.0.98.62:33985)
0620 04:30:33.373571 (+ 1987us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:33.373595 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:33.373645 (+ 50us) server_negotiation.cc:407] Connection header received
0620 04:30:33.374549 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:33.374576 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:33.375234 (+ 658us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:33.375580 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:33.375962 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.377116 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.378478 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.378999 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.379129 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:33.379972 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:33.379977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:33.379991 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:30:33.380412 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:33.380510 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:30:33.380533 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:33.380645 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:33.380804 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 04:30:33.381128 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1508,"thread_start_us":132,"threads_started":1}
I20260620 04:30:33.383912 5897 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9bca740ac3403fbc4473ec0c0122ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33985 } }
I20260620 04:30:33.384184 5897 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:33.384246 5897 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed9bca740ac3403fbc4473ec0c0122ca, State: Initialized, Role: FOLLOWER
I20260620 04:30:33.384709 5897 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9bca740ac3403fbc4473ec0c0122ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33985 } }
I20260620 04:30:33.384852 5897 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:33.384922 5897 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:33.385046 5897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:33.385058 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:30:33.415858 5897 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9bca740ac3403fbc4473ec0c0122ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33985 } }
I20260620 04:30:33.416353 5897 leader_election.cc:302] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed9bca740ac3403fbc4473ec0c0122ca; no voters:
I20260620 04:30:33.416839 5897 leader_election.cc:288] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:33.417200 5902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:33.417488 5902 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [term 1 LEADER]: Becoming Leader. State: Replica: ed9bca740ac3403fbc4473ec0c0122ca, State: Running, Role: LEADER
I20260620 04:30:33.418138 5902 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9bca740ac3403fbc4473ec0c0122ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33985 } }
I20260620 04:30:33.418506 5897 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:33.420146 5903 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed9bca740ac3403fbc4473ec0c0122ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9bca740ac3403fbc4473ec0c0122ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33985 } } }
I20260620 04:30:33.420308 5903 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:33.420553 5904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed9bca740ac3403fbc4473ec0c0122ca. Latest consensus state: current_term: 1 leader_uuid: "ed9bca740ac3403fbc4473ec0c0122ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9bca740ac3403fbc4473ec0c0122ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33985 } } }
I20260620 04:30:33.420653 5916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:33.421072 5904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:33.423815 5916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:33.429486 5916 catalog_manager.cc:1347] Generated new cluster ID: a9a51a9232704f36a8f4e2daf8d97951
I20260620 04:30:33.429570 5916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:33.439348 5916 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:33.440701 5916 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:33.448060 5916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed9bca740ac3403fbc4473ec0c0122ca: Generated new TSK 0
I20260620 04:30:33.449431 5916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:30:33.938390 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:33.470098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.0.98.62:33985)
0620 04:30:33.470227 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:33.470232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:33.470441 (+ 209us) server_negotiation.cc:407] Connection header received
0620 04:30:33.470615 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:33.470618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:33.470679 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:33.470805 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0620 04:30:33.471047 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.471844 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.472804 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.473186 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.473290 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:33.473637 (+ 347us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 04:30:33.937863 (+464226us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 04:30:33.938234 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59738: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 04:30:33.938498 5900 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59738: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 04:30:33.938494 5926 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:33.469997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33985 (local address 127.0.0.1:59738)
0620 04:30:33.470356 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:33.470368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:33.470471 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:33.470821 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:33.470824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:33.470832 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 04:30:33.470938 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:33.470943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.471984 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.471987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:33.472684 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:33.472694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:33.473293 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:33.473314 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:33.473400 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:33.473424 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 04:30:33.938129 (+464705us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 04:30:33.938325 (+ 196us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33985: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":94,"threads_started":1}
I20260620 04:30:33.939129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 5830
2026-06-20T04:30:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (772 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (128 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:34 dist-test-slave-rzqm krb5kdc[5938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:34 dist-test-slave-rzqm krb5kdc[5938](info): set up 2 sockets
Jun 20 04:30:34 dist-test-slave-rzqm krb5kdc[5938](info): commencing operation
krb5kdc: starting...
W20260620 04:30:35.849200 392 mini_kdc.cc:121] Time spent starting KDC: real 1.636s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:35 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:35.965945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45651
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:36.077035 5951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:36.077384 5951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:36.077426 5951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:36.081707 5951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:36.081779 5951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:36.081804 5951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:36.081821 5951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:36.081835 5951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:36.087361 5951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:36.088482 5951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:36.089558 5951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:36.098018 5959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:36.098114 5956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:36.098018 5957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:36.098408 5951 server_base.cc:1034] running on GCE node
I20260620 04:30:36.099035 5951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:36.100226 5951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:36.101379 5951 hybrid_clock.cc:648] HybridClock initialized: now 1781929836101349 us; error 46 us; skew 500 ppm
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:36.104282 5951 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:36.105302 5951 webserver.cc:460] Webserver started at http://127.0.98.62:46705/ using document root <none> and password file <none>
I20260620 04:30:36.105872 5951 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:36.105933 5951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:36.106108 5951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:36.108162 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "d39269f4c87c46a187fb14a9338f3404"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "0ffbef76f0c425858c593c7d79006eb8"
server_key_iv: "ea87c3910cf445a4282127c4b2e976dd"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.108680 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "d39269f4c87c46a187fb14a9338f3404"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "0ffbef76f0c425858c593c7d79006eb8"
server_key_iv: "ea87c3910cf445a4282127c4b2e976dd"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.112422 5951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:30:36.114949 5966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:36.116284 5951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:30:36.116390 5951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "d39269f4c87c46a187fb14a9338f3404"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "0ffbef76f0c425858c593c7d79006eb8"
server_key_iv: "ea87c3910cf445a4282127c4b2e976dd"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.116483 5951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:36.155043 5951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:36.158587 5951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:36.158777 5951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:36.166091 5951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45651
I20260620 04:30:36.166376 6018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45651 every 8 connection(s)
I20260620 04:30:36.167150 5951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:36.170233 6019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:36.172801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 5951
I20260620 04:30:36.172935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:36.173240 392 external_mini_cluster.cc:1428] Setting key 25d1c55cdaee0fafa6731657532a4492
I20260620 04:30:36.177167 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404: Bootstrap starting.
I20260620 04:30:36.179939 6019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:36.180744 6019 log.cc:824] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:36.183216 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404: No bootstrap required, opened a new log
I20260620 04:30:36.187387 6019 raft_consensus.cc:348] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39269f4c87c46a187fb14a9338f3404" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260620 04:30:36.187624 6019 raft_consensus.cc:374] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:36.187660 6019 raft_consensus.cc:729] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39269f4c87c46a187fb14a9338f3404, State: Initialized, Role: FOLLOWER
I20260620 04:30:36.188129 6019 consensus_queue.cc:260] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39269f4c87c46a187fb14a9338f3404" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260620 04:30:36.188194 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:36.174816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.0.98.62:45651)
0620 04:30:36.175286 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:36.175303 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:36.175343 (+ 40us) server_negotiation.cc:407] Connection header received
0620 04:30:36.176063 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:36.176090 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:36.176478 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:36.176797 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:36.177790 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.178877 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.180069 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.180624 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.180763 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:36.183607 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:36.183613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:36.183627 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:36.183654 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:36.185286 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:36.185796 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:36.185801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:36.185808 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:36.185921 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:36.186178 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:36.186180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:36.186181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:36.186589 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:36.186743 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:36.187020 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 04:30:36.187342 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":134,"threads_started":1}
I20260620 04:30:36.188256 6019 raft_consensus.cc:388] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:36.188308 6019 raft_consensus.cc:482] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:36.188402 6019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:36.189352 6019 raft_consensus.cc:504] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39269f4c87c46a187fb14a9338f3404" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260620 04:30:36.189667 6019 leader_election.cc:302] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d39269f4c87c46a187fb14a9338f3404; no voters:
I20260620 04:30:36.189977 6019 leader_election.cc:288] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:36.190052 6024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:36.190227 6024 raft_consensus.cc:686] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [term 1 LEADER]: Becoming Leader. State: Replica: d39269f4c87c46a187fb14a9338f3404, State: Running, Role: LEADER
I20260620 04:30:36.190587 6024 consensus_queue.cc:237] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39269f4c87c46a187fb14a9338f3404" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } }
I20260620 04:30:36.191282 6019 sys_catalog.cc:564] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:36.192176 6026 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d39269f4c87c46a187fb14a9338f3404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39269f4c87c46a187fb14a9338f3404" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } } }
I20260620 04:30:36.192369 6026 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:36.192480 6025 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d39269f4c87c46a187fb14a9338f3404. Latest consensus state: current_term: 1 leader_uuid: "d39269f4c87c46a187fb14a9338f3404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d39269f4c87c46a187fb14a9338f3404" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45651 } } }
I20260620 04:30:36.192633 6025 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:36.193429 6033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:36.196264 6033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:36.202131 6033 catalog_manager.cc:1347] Generated new cluster ID: 11bf883fc5174b6b8094136cc1e7c479
I20260620 04:30:36.202206 6033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:36.247318 6033 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:36.248971 6033 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:36.268275 6033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d39269f4c87c46a187fb14a9338f3404: Generated new TSK 0
I20260620 04:30:36.269361 6033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:30:36.304210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:36.416357 6045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:36.416625 6045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:36.416676 6045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:36.421073 6045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:36.421149 6045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:36.421226 6045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:36.426685 6045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:36.427721 6045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:36.428715 6045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:36.438350 6050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:36.438491 6045 server_base.cc:1034] running on GCE node
W20260620 04:30:36.438380 6053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:36.438355 6051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:36.439085 6045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:36.439769 6045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:36.440932 6045 hybrid_clock.cc:648] HybridClock initialized: now 1781929836440899 us; error 46 us; skew 500 ppm
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:36.443759 6045 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:30:36.444698 6045 webserver.cc:460] Webserver started at http://127.0.98.1:37003/ using document root <none> and password file <none>
I20260620 04:30:36.445393 6045 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:36.445475 6045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:36.445709 6045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:36.447839 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "93afbe4ed621428c9b258ff4fec64d72"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "82ab0988b4dec3dcb710a2b98e17c3ab"
server_key_iv: "6f4919fe43941273f209025a07092519"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.448334 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "93afbe4ed621428c9b258ff4fec64d72"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "82ab0988b4dec3dcb710a2b98e17c3ab"
server_key_iv: "6f4919fe43941273f209025a07092519"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.451859 6045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:30:36.454267 6060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:36.455505 6045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:36.455618 6045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "93afbe4ed621428c9b258ff4fec64d72"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "82ab0988b4dec3dcb710a2b98e17c3ab"
server_key_iv: "6f4919fe43941273f209025a07092519"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.455721 6045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:36.474180 6045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:36.477411 6045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:36.477586 6045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:36.478144 6045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:36.478998 6045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:36.479048 6045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:36.479089 6045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:36.479108 6045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:36.487219 6045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46617
I20260620 04:30:36.487269 6173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46617 every 8 connection(s)
I20260620 04:30:36.488121 6045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:36.490597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 6045
I20260620 04:30:36.490705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:36.490983 392 external_mini_cluster.cc:1428] Setting key a88123a29ef4e9f69d3a8893a43de981
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:30:36.501873 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:36.489854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60927 (local address 127.0.98.62:45651)
0620 04:30:36.490000 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:36.490004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:36.490961 (+ 957us) server_negotiation.cc:407] Connection header received
0620 04:30:36.492439 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:36.492443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:36.492493 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:36.492573 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:36.493825 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.494355 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.495556 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.495880 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.495979 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:36.498622 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:36.498625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:36.498627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:36.498652 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:36.499952 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:36.500523 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:36.500526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:36.500527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:36.500598 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:36.500894 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:36.500896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:36.500897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:36.501096 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:36.501168 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:36.501639 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 04:30:36.501748 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:30:36.502846 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:36.490318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.1:60927)
0620 04:30:36.490791 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:36.490846 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:36.492228 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:36.492737 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:36.492744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:36.493169 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:36.493675 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:36.493689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.494474 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.494477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:36.495423 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:36.495444 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.496036 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.496039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:36.496199 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:36.496776 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:36.496799 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:36.498441 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:36.500073 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:36.500079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:36.500097 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:36.500426 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:36.500681 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:36.500683 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:36.500685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:36.500827 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:36.501170 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:36.501175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:36.501437 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 04:30:36.501594 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 04:30:36.501922 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260620 04:30:36.504086 6174 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45651
I20260620 04:30:36.504384 6174 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:36.504958 6174 heartbeater.cc:511] Master 127.0.98.62:45651 requested a full tablet report, sending...
I20260620 04:30:36.506767 5979 ts_manager.cc:194] Registered new tserver with Master: 93afbe4ed621428c9b258ff4fec64d72 (127.0.98.1:46617)
I20260620 04:30:36.508062 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:30:36.517179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:36.630120 6179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:36.630431 6179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:36.630494 6179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:36.634812 6179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:36.634898 6179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:36.634982 6179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:36.640507 6179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:36.641660 6179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:36.642706 6179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:36.652302 6184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:36.652339 6179 server_base.cc:1034] running on GCE node
W20260620 04:30:36.652287 6185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:36.652287 6187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:36.653111 6179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:36.653931 6179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:36.655170 6179 hybrid_clock.cc:648] HybridClock initialized: now 1781929836655089 us; error 104 us; skew 500 ppm
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:36.658073 6179 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:30:36.659036 6179 webserver.cc:460] Webserver started at http://127.0.98.2:41317/ using document root <none> and password file <none>
I20260620 04:30:36.659634 6179 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:36.659703 6179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:36.659902 6179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:36.662046 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "256c83a57c034094a8d7561ac90ca716"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "8f2a262b46a71bfaaf42f9401efaa8e3"
server_key_iv: "e88f8e650f026fc0e77572edb042146b"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.662590 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "256c83a57c034094a8d7561ac90ca716"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "8f2a262b46a71bfaaf42f9401efaa8e3"
server_key_iv: "e88f8e650f026fc0e77572edb042146b"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.666227 6179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:30:36.668704 6194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:36.670017 6179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:36.670128 6179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "256c83a57c034094a8d7561ac90ca716"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "8f2a262b46a71bfaaf42f9401efaa8e3"
server_key_iv: "e88f8e650f026fc0e77572edb042146b"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.670229 6179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:36.684244 6179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:36.687690 6179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:36.687888 6179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:36.688527 6179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:36.689464 6179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:36.689514 6179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:36.689569 6179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:36.689597 6179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:36.698698 6179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37335
I20260620 04:30:36.698778 6307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37335 every 8 connection(s)
I20260620 04:30:36.699625 6179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:36.704344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 6179
I20260620 04:30:36.704461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:36.704743 392 external_mini_cluster.cc:1428] Setting key a5000c016c8d31d08568d36a34d082c9
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:30:36.714907 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:36.701442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57387 (local address 127.0.98.62:45651)
0620 04:30:36.701585 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:36.701590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:36.702551 (+ 961us) server_negotiation.cc:407] Connection header received
0620 04:30:36.703929 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:36.703933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:36.703982 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:36.704059 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:36.705712 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.706414 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.708077 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.708437 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.708554 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:36.711283 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:36.711286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:36.711288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:36.711313 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:36.712724 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:36.713295 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:36.713298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:36.713299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:36.713379 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:36.713746 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:36.713749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:36.713750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:36.713946 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:36.714033 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:36.714597 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 04:30:36.714728 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:30:36.715739 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:36.701929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.2:57387)
0620 04:30:36.702390 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:36.702438 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:36.703673 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:36.704266 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:36.704278 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:36.704834 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:36.705532 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:36.705554 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.706547 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.706551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:36.707924 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:36.707937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.708612 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.708615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:36.708778 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:36.709401 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:36.709422 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:36.711085 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:36.712837 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:36.712844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:36.712863 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:36.713197 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:36.713462 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:36.713465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:36.713467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:36.713649 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:36.714045 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:36.714052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:36.714320 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 04:30:36.714528 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 04:30:36.714870 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:30:36.716924 6308 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45651
I20260620 04:30:36.717214 6308 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:36.717792 6308 heartbeater.cc:511] Master 127.0.98.62:45651 requested a full tablet report, sending...
I20260620 04:30:36.718981 5979 ts_manager.cc:194] Registered new tserver with Master: 256c83a57c034094a8d7561ac90ca716 (127.0.98.2:37335)
I20260620 04:30:36.719630 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:30:36.733595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:36.845048 6313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:36.845474 6313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:36.845544 6313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:36.851658 6313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:36.851723 6313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:36.851785 6313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:36.857407 6313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:36.858501 6313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:36.859576 6313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:36.869231 6319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:36.869220 6318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:36.869314 6313 server_base.cc:1034] running on GCE node
W20260620 04:30:36.869225 6321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:36.869859 6313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:36.870543 6313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:36.871701 6313 hybrid_clock.cc:648] HybridClock initialized: now 1781929836871660 us; error 56 us; skew 500 ppm
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:36.874512 6313 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:30:36.875409 6313 webserver.cc:460] Webserver started at http://127.0.98.3:39645/ using document root <none> and password file <none>
I20260620 04:30:36.875957 6313 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:36.876017 6313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:36.876194 6313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:36.878311 6313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "4fd717c63ec6446886e855f3304495e9"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "b0add17fd65a720f32deaf62e3e510b3"
server_key_iv: "be9d024720c672228b3504338f9c1c84"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.878835 6313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4fd717c63ec6446886e855f3304495e9"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "b0add17fd65a720f32deaf62e3e510b3"
server_key_iv: "be9d024720c672228b3504338f9c1c84"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.882723 6313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:30:36.885502 6328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:36.886914 6313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:36.887043 6313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "4fd717c63ec6446886e855f3304495e9"
format_stamp: "Formatted at 2026-06-20 04:30:36 on dist-test-slave-rzqm"
server_key: "b0add17fd65a720f32deaf62e3e510b3"
server_key_iv: "be9d024720c672228b3504338f9c1c84"
server_key_version: "encryptionkey@0"
I20260620 04:30:36.887166 6313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:36.912569 6313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:36.915820 6313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:36.916002 6313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:36.916615 6313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:36.917523 6313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:36.917569 6313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:36.917613 6313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:36.917642 6313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:36.926640 6313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39267
I20260620 04:30:36.926661 6441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39267 every 8 connection(s)
I20260620 04:30:36.927585 6313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:36.930336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 6313
I20260620 04:30:36.930449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:36.930732 392 external_mini_cluster.cc:1428] Setting key 9a87fb55fc70582518f48548c9cf3a99
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:36.941622 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:36.929434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45285 (local address 127.0.98.62:45651)
0620 04:30:36.929562 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:36.929565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:36.930475 (+ 910us) server_negotiation.cc:407] Connection header received
0620 04:30:36.932120 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:36.932123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:36.932168 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:36.932249 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:36.933636 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.934178 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.935459 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.935677 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.935757 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:36.938424 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:36.938428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:36.938430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:36.938456 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:36.939692 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:36.940184 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:36.940187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:36.940188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:36.940272 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:36.940586 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:36.940589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:36.940591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:36.940784 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:36.940882 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:36.941392 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 04:30:36.941498 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:30:36.942502 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:36.929815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45651 (local address 127.0.98.3:45285)
0620 04:30:36.930317 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:36.930364 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:36.931921 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:36.932396 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:36.932403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:36.932953 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:36.933498 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:36.933513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.934318 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.934324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:36.935331 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:36.935342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.935896 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.935899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:36.936071 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:36.936665 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:36.936682 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:36.938234 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:36.939807 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:36.939811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:36.939825 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:36.940084 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:36.940347 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:36.940349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:36.940350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:36.940483 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:36.940879 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:36.940884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:36.941181 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 04:30:36.941345 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 04:30:36.941611 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":114,"threads_started":1}
I20260620 04:30:36.943883 6442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45651
I20260620 04:30:36.944190 6442 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:36.944802 6442 heartbeater.cc:511] Master 127.0.98.62:45651 requested a full tablet report, sending...
I20260620 04:30:36.945890 5979 ts_manager.cc:194] Registered new tserver with Master: 4fd717c63ec6446886e855f3304495e9 (127.0.98.3:39267)
I20260620 04:30:36.946497 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45285
I20260620 04:30:36.956676 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/test-admin.keytab.
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:36.971918 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 04:30:36.975057 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:36.973528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.0.98.1:46617)
0620 04:30:36.973855 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:36.973870 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:36.973892 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:30:36.973942 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:36.973947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:36.974135 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:36.974273 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:36.974879 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49968: BlockingRecv error: recv got EOF from 127.0.0.1:49968 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":127,"threads_started":1}
W20260620 04:30:36.975133 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:36.976172 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:36.977123 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:30:36.987389 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:36.978162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.0.98.1:46617)
0620 04:30:36.978295 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:36.978298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:36.978360 (+ 62us) server_negotiation.cc:407] Connection header received
0620 04:30:36.978478 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:36.978480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:36.978522 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:36.978591 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:36.979150 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.979966 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.980849 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.981316 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.981430 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:36.983731 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:36.983741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:36.983756 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:36.983783 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:36.985386 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:36.985912 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:36.985919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:36.985920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:36.986014 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:36.986313 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:36.986316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:36.986318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:36.986614 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:36.986797 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:36.987036 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 04:30:36.987199 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 04:30:36.991814 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:36.990673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.0.98.1:46617)
0620 04:30:36.990804 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:36.990808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:36.990880 (+ 72us) server_negotiation.cc:407] Connection header received
0620 04:30:36.991040 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:36.991044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:36.991087 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:36.991155 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:36.991672 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49986: BlockingRecv error: recv got EOF from 127.0.0.1:49986 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 04:30:36.991873 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:36.993156 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:36.994204 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 04:30:36 dist-test-slave-rzqm krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:30:37.003217 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:36.995138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.0.98.1:46617)
0620 04:30:36.995285 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:36.995289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:36.995343 (+ 54us) server_negotiation.cc:407] Connection header received
0620 04:30:36.995460 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:36.995462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:36.995503 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:36.995570 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:36.996004 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.996549 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.997432 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:36.997666 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:36.997750 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:37.000062 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:37.000065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:37.000068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:37.000096 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:37.001491 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:37.001934 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:37.001937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:37.001938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:37.002028 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:37.002309 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:37.002311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:37.002313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:37.002494 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:37.002683 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:37.002922 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 04:30:37.003052 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 04:30:37.006309 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:37.005206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.0.98.1:46617)
0620 04:30:37.005338 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:37.005342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:37.005434 (+ 92us) server_negotiation.cc:407] Connection header received
0620 04:30:37.005589 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:37.005592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:37.005634 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:37.005713 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:37.006206 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49994: BlockingRecv error: recv got EOF from 127.0.0.1:49994 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 04:30:37.006482 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781929807654528-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 04:30:37 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:37.007594 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 04:30:37 dist-test-slave-rzqm krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:37.008591 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 04:30:37 dist-test-slave-rzqm krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:30:37.018080 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:37.009599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.0.98.1:46617)
0620 04:30:37.009745 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:37.009749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:37.009765 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:30:37.009918 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:37.009921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:37.009963 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:37.010035 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:37.010591 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:37.011102 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:37.011939 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:37.012178 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:37.012269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:37.014810 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:37.014813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:37.014815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:37.014855 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:37.016228 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:37.016668 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:37.016671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:37.016672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:37.016757 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:37.017052 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:37.017056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:37.017058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:37.017240 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:37.017341 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:37.017704 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 04:30:37.017814 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 04:30:37.019250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 6045
I20260620 04:30:37.024225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 6179
I20260620 04:30:37.028990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 6313
I20260620 04:30:37.033528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 5951
2026-06-20T04:30:37Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2840 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:37 dist-test-slave-rzqm krb5kdc[6462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:37 dist-test-slave-rzqm krb5kdc[6462](info): set up 2 sockets
Jun 20 04:30:37 dist-test-slave-rzqm krb5kdc[6462](info): commencing operation
krb5kdc: starting...
W20260620 04:30:37.985323 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:38 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 04:30:38.103293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37507
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:38.215745 6475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:38.216037 6475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:38.216095 6475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:38.220259 6475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:38.220335 6475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:38.220363 6475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:38.220386 6475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:38.220423 6475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:38.225958 6475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:38.227244 6475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:38.228339 6475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:38.237718 6480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:38.237718 6483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:38.237768 6481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:38.238068 6475 server_base.cc:1034] running on GCE node
I20260620 04:30:38.238801 6475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:38.240098 6475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:38.241300 6475 hybrid_clock.cc:648] HybridClock initialized: now 1781929838241264 us; error 53 us; skew 500 ppm
Jun 20 04:30:38 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:38.247526 6475 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 04:30:38.248625 6475 webserver.cc:460] Webserver started at http://127.0.98.62:37557/ using document root <none> and password file <none>
I20260620 04:30:38.249291 6475 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:38.249366 6475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:38.249554 6475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:38.251801 6475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "0c6740c79de24b00b6ec13d810e271bc"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "ba0bde0ea601b18ae1df2845f41f75f3"
server_key_iv: "ad3ee6e5079ae6b1d2e00dd876a03156"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.252373 6475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c6740c79de24b00b6ec13d810e271bc"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "ba0bde0ea601b18ae1df2845f41f75f3"
server_key_iv: "ad3ee6e5079ae6b1d2e00dd876a03156"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.256369 6475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:30:38.259127 6490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:38.260419 6475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 04:30:38.260541 6475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "0c6740c79de24b00b6ec13d810e271bc"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "ba0bde0ea601b18ae1df2845f41f75f3"
server_key_iv: "ad3ee6e5079ae6b1d2e00dd876a03156"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.260644 6475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:38.289363 6475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:38.292420 6475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:38.292587 6475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:38.299342 6475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37507
I20260620 04:30:38.299602 6542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37507 every 8 connection(s)
I20260620 04:30:38.300321 6475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:38.303079 6543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:38.308547 6543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: Bootstrap starting.
I20260620 04:30:38.310071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 6475
I20260620 04:30:38.310204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:38.310483 392 external_mini_cluster.cc:1428] Setting key 9021f4248c2b9ba0cbf5026fde355fd9
I20260620 04:30:38.311466 6543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:38.312278 6543 log.cc:824] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:38.314687 6543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: No bootstrap required, opened a new log
I20260620 04:30:38.318920 6543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } }
I20260620 04:30:38.319242 6543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:38.319314 6543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6740c79de24b00b6ec13d810e271bc, State: Initialized, Role: FOLLOWER
Jun 20 04:30:38 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:30:38.319861 6543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } }
I20260620 04:30:38.320021 6543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:38.320097 6543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:38.320199 6543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:38.321195 6543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } }
I20260620 04:30:38.321534 6543 leader_election.cc:302] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c6740c79de24b00b6ec13d810e271bc; no voters:
I20260620 04:30:38.321837 6543 leader_election.cc:288] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:38.321969 6548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:38.322182 6548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 1 LEADER]: Becoming Leader. State: Replica: 0c6740c79de24b00b6ec13d810e271bc, State: Running, Role: LEADER
I20260620 04:30:38.322590 6548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } }
I20260620 04:30:38.323122 6543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:38.323889 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c6740c79de24b00b6ec13d810e271bc. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } }
I20260620 04:30:38.324622 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:38.324512 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } }
I20260620 04:30:38.324981 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:38.326848 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:38.311663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.0.98.62:37507)
0620 04:30:38.312111 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:38.312128 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:38.312167 (+ 39us) server_negotiation.cc:407] Connection header received
0620 04:30:38.313065 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:38.313094 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:38.313484 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:38.313793 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:38.314713 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:38.315775 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:38.317064 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:38.317605 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:38.317729 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:38.320595 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:38.320601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:38.320616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:38.320647 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:38.322741 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:38.323299 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:38.323305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:38.323312 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:38.323454 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:38.323805 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:38.323809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:38.323810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:38.324371 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:38.324570 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:38.325534 (+ 964us) server_negotiation.cc:299] Negotiation successful
0620 04:30:38.325875 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
W20260620 04:30:38.328469 6563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:30:38.328526 6563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:30:38.328619 6562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:38.329568 6562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:38.335060 6562 catalog_manager.cc:1347] Generated new cluster ID: db6e4e50e6564cd596f8c627a50e4c1b
I20260620 04:30:38.335148 6562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:38.349668 6562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:38.350783 6562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:38.355577 6562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: Generated new TSK 0
I20260620 04:30:38.356463 6562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 04:30:38.388183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37507
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:38.499042 6569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:38.499320 6569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:38.499362 6569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:38.503360 6569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:38.503424 6569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:38.503485 6569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:38.508791 6569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:38.509939 6569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:38.511029 6569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:38.522940 6574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:38.526314 6575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:38.526378 6569 server_base.cc:1034] running on GCE node
W20260620 04:30:38.526616 6577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:38.527204 6569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:38.528046 6569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:38.529242 6569 hybrid_clock.cc:648] HybridClock initialized: now 1781929838529214 us; error 46 us; skew 500 ppm
Jun 20 04:30:38 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:38.531967 6569 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:30:38.532912 6569 webserver.cc:460] Webserver started at http://127.0.98.1:46753/ using document root <none> and password file <none>
I20260620 04:30:38.533486 6569 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:38.533552 6569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:38.533742 6569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:38.535794 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "bac4c91bd5224017a643817100960103"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "c64f53b7afaf05c8c40be11df6a4bd8c"
server_key_iv: "10ca286827b247841be430a0952b6d5c"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.536338 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bac4c91bd5224017a643817100960103"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "c64f53b7afaf05c8c40be11df6a4bd8c"
server_key_iv: "10ca286827b247841be430a0952b6d5c"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.540035 6569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:30:38.542450 6584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:38.543642 6569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:38.543746 6569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "bac4c91bd5224017a643817100960103"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "c64f53b7afaf05c8c40be11df6a4bd8c"
server_key_iv: "10ca286827b247841be430a0952b6d5c"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.543849 6569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:38.569204 6569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:38.572440 6569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:38.572630 6569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:38.573262 6569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:38.574121 6569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:38.574169 6569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:38.574222 6569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:38.574251 6569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:38.582798 6569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45291
I20260620 04:30:38.582820 6697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45291 every 8 connection(s)
I20260620 04:30:38.583663 6569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:38.584740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 6569
I20260620 04:30:38.584863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:38.585156 392 external_mini_cluster.cc:1428] Setting key ec65799d85852fe2ee21cb37dc8e97a6
Jun 20 04:30:38 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 04:30:38.598001 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:38.585527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51553 (local address 127.0.98.62:37507)
0620 04:30:38.585657 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:38.585662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:38.587154 (+ 1492us) server_negotiation.cc:407] Connection header received
0620 04:30:38.588262 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:38.588265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:38.588323 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:38.588406 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:38.589772 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:38.590302 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:38.591488 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:38.591732 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:38.591820 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:38.594417 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:38.594420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:38.594422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:38.594447 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:38.595945 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:38.596465 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:38.596468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:38.596470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:38.596568 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:38.596881 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:38.596884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:38.596886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:38.597123 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:38.597212 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:38.597703 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 04:30:38.597833 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:30:38.598829 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:38.585988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.1:51553)
0620 04:30:38.586951 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:38.587010 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:38.588007 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:38.588571 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:38.588579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:38.588933 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:38.589620 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:38.589639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:38.590431 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:38.590434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:38.591359 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:38.591369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:38.591878 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:38.591882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:38.592053 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:38.592613 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:38.592630 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:38.594244 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:38.596084 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:38.596090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:38.596107 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:38.596363 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:38.596655 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:38.596658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:38.596661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:38.596796 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:38.597217 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:38.597224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:38.597474 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 04:30:38.597646 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 04:30:38.597988 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":593,"threads_started":1}
I20260620 04:30:38.600086 6698 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37507
I20260620 04:30:38.600363 6698 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:38.600983 6698 heartbeater.cc:511] Master 127.0.98.62:37507 requested a full tablet report, sending...
I20260620 04:30:38.602602 6507 ts_manager.cc:194] Registered new tserver with Master: bac4c91bd5224017a643817100960103 (127.0.98.1:45291)
I20260620 04:30:38.603678 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 04:30:38.609305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37507
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:38.719430 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:38.719688 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:38.719731 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:38.723753 6703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:38.723814 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:38.723871 6703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:38.729158 6703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:38.730180 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:38.731181 6703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:38.740468 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:38.740507 6709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:38.740468 6711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:38.740541 6703 server_base.cc:1034] running on GCE node
I20260620 04:30:38.741057 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:38.741715 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:38.742872 6703 hybrid_clock.cc:648] HybridClock initialized: now 1781929838742844 us; error 37 us; skew 500 ppm
Jun 20 04:30:38 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:38.745684 6703 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:30:38.746631 6703 webserver.cc:460] Webserver started at http://127.0.98.2:35219/ using document root <none> and password file <none>
I20260620 04:30:38.747228 6703 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:38.747294 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:38.747545 6703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:38.749842 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "50af5eaac37b4e4690ea841a0a67096a"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "24f871ce64617dd7e04041134981aac4"
server_key_iv: "a23cbc724901e61c3e7e4b032e8089f3"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.750385 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "50af5eaac37b4e4690ea841a0a67096a"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "24f871ce64617dd7e04041134981aac4"
server_key_iv: "a23cbc724901e61c3e7e4b032e8089f3"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.754189 6703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 04:30:38.756757 6718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:38.758167 6703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:38.758328 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "50af5eaac37b4e4690ea841a0a67096a"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "24f871ce64617dd7e04041134981aac4"
server_key_iv: "a23cbc724901e61c3e7e4b032e8089f3"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.758462 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:38.775466 6703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:38.778683 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:38.778869 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:38.779474 6703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:38.780293 6703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:38.780339 6703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:38.780395 6703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:38.780423 6703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:38.788719 6703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35869
I20260620 04:30:38.788739 6831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35869 every 8 connection(s)
I20260620 04:30:38.789660 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:38.795275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 6703
I20260620 04:30:38.795377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:38.795625 392 external_mini_cluster.cc:1428] Setting key 0ed25be44e4b57fdca6a6b3963ab80ee
Jun 20 04:30:38 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:30:38.803673 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:38.791439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60935 (local address 127.0.98.62:37507)
0620 04:30:38.791560 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:38.791563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:38.792332 (+ 769us) server_negotiation.cc:407] Connection header received
0620 04:30:38.793666 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:38.793669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:38.793722 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:38.793803 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:38.795022 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:38.795602 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:38.796942 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:38.797232 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:38.797351 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:38.799953 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:38.799957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:38.799960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:38.799988 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:38.801533 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:38.802057 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:38.802061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:38.802063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:38.802158 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:38.802510 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:38.802521 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:38.802524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:38.802764 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:38.802920 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:38.803376 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 04:30:38.803503 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:30:38.804450 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:38.791746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.2:60935)
0620 04:30:38.792168 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:38.792213 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:38.793425 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:38.794017 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:38.794025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:38.794383 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:38.794871 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:38.794884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:38.795726 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:38.795730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:38.796827 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:38.796838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:38.797378 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:38.797383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:38.797569 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:38.798114 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:38.798131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:38.799774 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:38.801653 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:38.801658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:38.801672 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:38.801939 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:38.802269 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:38.802272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:38.802273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:38.802421 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:38.802871 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:38.802876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:38.803132 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 04:30:38.803292 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 04:30:38.803574 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:30:38.805646 6832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37507
I20260620 04:30:38.805907 6832 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:38.806372 6832 heartbeater.cc:511] Master 127.0.98.62:37507 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260620 04:30:38.807443 6507 ts_manager.cc:194] Registered new tserver with Master: 50af5eaac37b4e4690ea841a0a67096a (127.0.98.2:35869)
I20260620 04:30:38.808131 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 04:30:38.821988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37507
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:38.932785 6837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:38.933120 6837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:38.933187 6837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:38.937256 6837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:38.937321 6837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:38.937403 6837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:38.942713 6837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:38.943794 6837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:38.944783 6837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:38.954483 6842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:38.954488 6843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:38.954572 6845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:38.954543 6837 server_base.cc:1034] running on GCE node
I20260620 04:30:38.955129 6837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:38.955808 6837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:38.956982 6837 hybrid_clock.cc:648] HybridClock initialized: now 1781929838956957 us; error 43 us; skew 500 ppm
Jun 20 04:30:38 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:38.959741 6837 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:30:38.960656 6837 webserver.cc:460] Webserver started at http://127.0.98.3:40785/ using document root <none> and password file <none>
I20260620 04:30:38.961231 6837 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:38.961294 6837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:38.961462 6837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:38.963550 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "2919829f57ac4813af09f80a642e8f97"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "4bff71da1798846f8423844e6b69369c"
server_key_iv: "43328aeb8136fd5570bd4644182468e0"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.964089 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2919829f57ac4813af09f80a642e8f97"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "4bff71da1798846f8423844e6b69369c"
server_key_iv: "43328aeb8136fd5570bd4644182468e0"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.967958 6837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 04:30:38.970535 6852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:38.971840 6837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:38.971957 6837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "2919829f57ac4813af09f80a642e8f97"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "4bff71da1798846f8423844e6b69369c"
server_key_iv: "43328aeb8136fd5570bd4644182468e0"
server_key_version: "encryptionkey@0"
I20260620 04:30:38.972072 6837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:38.988831 6837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:38.992156 6837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:38.992340 6837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:38.992945 6837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:38.993824 6837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:38.993870 6837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:38.993908 6837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:38.993932 6837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.002326 6837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42819
I20260620 04:30:39.002358 6965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42819 every 8 connection(s)
I20260620 04:30:39.003216 6837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:39.008306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 6837
I20260620 04:30:39.008401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:39.008653 392 external_mini_cluster.cc:1428] Setting key 61d55bf03db2ae45ae09ae6441431cb6
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:30:39.016986 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.005042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46887 (local address 127.0.98.62:37507)
0620 04:30:39.005173 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.005177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.006082 (+ 905us) server_negotiation.cc:407] Connection header received
0620 04:30:39.007375 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.007379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.007428 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.007503 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.008732 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.009435 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.010987 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.011234 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.011332 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.013702 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.013705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.013707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.013734 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.015098 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.015548 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.015550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.015551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.015631 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.015959 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.015967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.015968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.016176 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.016271 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.016742 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.016864 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:30:39.017850 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.005469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.3:46887)
0620 04:30:39.005918 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.005962 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.007160 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.007677 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.007684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.008057 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.008587 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.008603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.009569 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.009572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.010853 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.010864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.011373 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.011376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.011521 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.012066 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.012083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.013535 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.015200 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.015205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.015218 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.015468 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.015712 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.015714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.015716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.015847 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.016264 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.016268 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.016502 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.016664 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.016927 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260620 04:30:39.018997 6966 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37507
I20260620 04:30:39.019213 6966 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:39.019654 6966 heartbeater.cc:511] Master 127.0.98.62:37507 requested a full tablet report, sending...
I20260620 04:30:39.020573 6507 ts_manager.cc:194] Registered new tserver with Master: 2919829f57ac4813af09f80a642e8f97 (127.0.98.3:42819)
I20260620 04:30:39.021202 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46887
I20260620 04:30:39.022579 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 04:30:39.045979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45881
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37507,127.0.98.61:45881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:39.155601 6971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:39.155900 6971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:39.155958 6971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:39.160153 6971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:39.160233 6971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:39.160259 6971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:39.160281 6971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:39.160307 6971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:39.165903 6971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37507,127.0.98.61:45881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
W20260620 04:30:39.166149 6971 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:37507,127.0.98.61:45881'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 04:30:39.167080 6971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:39.168097 6971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:39.176329 6977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:39.176323 6979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:39.176329 6976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:39.176710 6971 server_base.cc:1034] running on GCE node
I20260620 04:30:39.177310 6971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:39.178403 6971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:39.179620 6971 hybrid_clock.cc:648] HybridClock initialized: now 1781929839179601 us; error 31 us; skew 500 ppm
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:39.182531 6971 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 04:30:39.183481 6971 webserver.cc:460] Webserver started at http://127.0.98.61:32921/ using document root <none> and password file <none>
I20260620 04:30:39.184026 6971 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:39.184085 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:39.184274 6971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:39.186384 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data/instance:
uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c"
format_stamp: "Formatted at 2026-06-20 04:30:39 on dist-test-slave-rzqm"
server_key: "640e965b7a8ffa458bfca09eefe3444e"
server_key_iv: "0341b42a9eb7c472fe65c66fdb980785"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.186911 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal/instance:
uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c"
format_stamp: "Formatted at 2026-06-20 04:30:39 on dist-test-slave-rzqm"
server_key: "640e965b7a8ffa458bfca09eefe3444e"
server_key_iv: "0341b42a9eb7c472fe65c66fdb980785"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.190487 6971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:30:39.192940 6986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.194254 6971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:39.194375 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal
uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c"
format_stamp: "Formatted at 2026-06-20 04:30:39 on dist-test-slave-rzqm"
server_key: "640e965b7a8ffa458bfca09eefe3444e"
server_key_iv: "0341b42a9eb7c472fe65c66fdb980785"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.194478 6971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:39.212824 6971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:39.216058 6971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:39.216284 6971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:39.223109 6971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45881
I20260620 04:30:39.223349 7038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45881 every 8 connection(s)
I20260620 04:30:39.224160 6971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:30:39.226969 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:39.230684 7039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:30:39.232091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 6971
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:30:39.244144 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.232544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.0.98.62:37507)
0620 04:30:39.232697 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.232702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.233647 (+ 945us) server_negotiation.cc:407] Connection header received
0620 04:30:39.234537 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.234539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.234583 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.234649 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.235544 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.236061 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.237193 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.237508 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.237583 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.240399 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.240402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.240404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.240427 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.241906 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.242470 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.242473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.242475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.242567 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.242887 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.242890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.242891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.243165 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.243254 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.243852 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.243971 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:30:39.245110 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.232971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:41794)
0620 04:30:39.233493 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.233543 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.234328 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.234718 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.234724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.235153 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.235408 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.235423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.236184 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.236187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.237092 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.237101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.237652 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.237661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.237969 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.238620 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.238636 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.240257 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.242018 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.242026 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.242037 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.242380 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.242660 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.242662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.242663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.242796 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.243256 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.243262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.243576 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.243737 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.244058 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":40,"spinlock_wait_cycles":4864,"thread_start_us":144,"threads_started":1}
I20260620 04:30:39.245110 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.232971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.0.98.61:45881)
0620 04:30:39.233513 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.233531 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.233574 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:30:39.234317 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.234341 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.234592 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.234760 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.235522 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.236514 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.237508 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.238104 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.238215 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.240796 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.240802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.240815 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.240840 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.242450 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.242859 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.242865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.242867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.242972 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.243236 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.243239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.243241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.243568 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.243757 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.243999 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.244112 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":418,"spinlock_wait_cycles":79744,"thread_start_us":170,"threads_started":1}
I20260620 04:30:39.246459 7039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:30:39.254213 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.246976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45881 (local address 127.0.0.1:49026)
0620 04:30:39.247167 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.247179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.247268 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.247606 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.247608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.247735 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.247823 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.247828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.248630 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.248633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.249192 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.249200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.249642 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.249645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.249718 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.250155 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.250166 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.251370 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.253040 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.253044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.253046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.253233 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.253454 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.253456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.253458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.253523 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.253836 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.253837 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.253917 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.253992 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.254091 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260620 04:30:39.254316 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.247039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.0.98.61:45881)
0620 04:30:39.247145 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.247149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.247453 (+ 304us) server_negotiation.cc:407] Connection header received
0620 04:30:39.247486 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.247488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.247520 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.247580 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.247937 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.248525 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.249311 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.249536 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.249621 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.251521 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.251524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.251526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.251546 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.252909 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.253319 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.253322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.253323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.253393 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.253585 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.253588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.253589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.253768 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.253844 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.254065 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.254167 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:30:39.257537 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Bootstrap starting.
I20260620 04:30:39.259940 7039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:39.260671 7039 log.cc:824] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:39.262689 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: No bootstrap required, opened a new log
I20260620 04:30:39.265913 7039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } }
I20260620 04:30:39.266131 7039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:39.266194 7039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eec4f1aa71e4e1585702f0c51bb0a9c, State: Initialized, Role: FOLLOWER
I20260620 04:30:39.266629 7039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } }
I20260620 04:30:39.267095 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } } }
I20260620 04:30:39.267232 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:30:39.267920 7039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 04:30:39.272557 7056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:30:39.272663 7056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:30:39.275307 6971 master_runner.cc:428] Detected that this master 7eec4f1aa71e4e1585702f0c51bb0a9c is joining an existing cluster
I20260620 04:30:39.275385 6971 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45881
I20260620 04:30:39.286330 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.278538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.0.98.61:45881)
0620 04:30:39.278664 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.278669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.278977 (+ 308us) server_negotiation.cc:407] Connection header received
0620 04:30:39.279099 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.279102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.279137 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.279196 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.279879 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.280371 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.281321 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.281644 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.281707 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.283035 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.283038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.283041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.283063 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.284734 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.285206 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.285210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.285212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.285284 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.285526 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.285529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.285531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.285731 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.285813 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.286044 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.286163 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":896}
I20260620 04:30:39.286899 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.278478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45881 (local address 127.0.0.1:49036)
0620 04:30:39.278911 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.278922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.279000 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.279218 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.279220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.279556 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.279757 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.279766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.280468 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.280470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.281200 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.281209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.281855 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.281858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.281925 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.282469 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.282481 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.282917 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.284846 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.284849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.284851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.285116 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.285368 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.285371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.285373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.285453 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.285813 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.285816 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.285900 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.286600 (+ 700us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.286735 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":39,"thread_start_us":78,"threads_started":1}
I20260620 04:30:39.287475 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.277964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:41802)
0620 04:30:39.278801 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.278816 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.278928 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.279312 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.279314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.279561 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.279765 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.279771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.280829 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.280832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.281682 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.281688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.282663 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.282668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.282763 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.283234 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.283246 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.283738 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.285659 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.285662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.285665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.285916 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.286207 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.286210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.286212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.286309 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.286914 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.286917 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.287022 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.287159 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.287301 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":41,"spinlock_wait_cycles":8960,"thread_start_us":160,"threads_started":1}
I20260620 04:30:39.287487 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.278096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.0.98.62:37507)
0620 04:30:39.278236 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.278241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.278898 (+ 657us) server_negotiation.cc:407] Connection header received
0620 04:30:39.279061 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.279065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.279125 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.279217 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.279891 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.280709 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.281807 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.282520 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.282625 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.283901 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.283905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.283908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.283937 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.285530 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.286017 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.286021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.286023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.286113 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.286401 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.286404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.286406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.286816 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.286920 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.287227 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.287347 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:30:39.304147 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.292497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.0.98.61:45881)
0620 04:30:39.292618 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.292622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.293017 (+ 395us) server_negotiation.cc:407] Connection header received
0620 04:30:39.293218 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.293221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.293275 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.293366 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.293957 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.294735 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.295948 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.296168 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.296253 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.298687 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.298691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.298693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.298719 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.302255 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.302813 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.302816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.302818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.302904 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.303178 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.303182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.303184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.303395 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.303460 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.303874 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.303991 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:30:39.305080 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.292439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45881 (local address 127.0.0.1:49052)
0620 04:30:39.292858 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.292914 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.293074 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.293392 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.293397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.293750 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.293858 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.293864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.294855 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.294858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.295820 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.295831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.296265 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.296268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.296396 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.296902 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.296914 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.298548 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.302379 (+ 3831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.302392 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.302398 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.302718 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.303005 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.303008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.303010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.303096 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.303536 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.303542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.303668 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.304675 (+ 1007us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.304882 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":73,"threads_started":1}
I20260620 04:30:39.306869 6507 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45881: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45881
}
attrs {
promote: true
}
}
dest_uuid: "0c6740c79de24b00b6ec13d810e271bc"
cas_config_opid_index: -1
I20260620 04:30:39.307534 6507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 04:30:39.310243 7066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7eec4f1aa71e4e1585702f0c51bb0a9c (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } } }
I20260620 04:30:39.310534 7013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:39.312014 7066 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45881
I20260620 04:30:39.312426 7013 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Refusing update from remote peer 0c6740c79de24b00b6ec13d810e271bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 04:30:39.312605 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } } }
I20260620 04:30:39.312772 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:39.312819 6971 master.cc:537] Master@127.0.98.61:45881 shutting down...
I20260620 04:30:39.313401 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 04:30:39.315594 6491 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 04:30:39.316583 6491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc -> Peer 7eec4f1aa71e4e1585702f0c51bb0a9c (127.0.98.61:45881): Couldn't send request to peer 7eec4f1aa71e4e1585702f0c51bb0a9c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260620 04:30:39.317600 6971 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:39.317901 6971 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:39.318012 6971 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: stopping tablet replica
I20260620 04:30:39.322667 6971 master.cc:552] Master@127.0.98.61:45881 shutdown complete.
I20260620 04:30:39.322721 6971 master_runner.cc:305] Clearing existing system tablet
I20260620 04:30:39.322933 6971 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:39.322980 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:39.325304 6971 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:39.326102 7072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.326339 6971 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:39.326467 6971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:39.326508 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal
uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c"
format_stamp: "Formatted at 2026-06-20 04:30:39 on dist-test-slave-rzqm"
server_key: "640e965b7a8ffa458bfca09eefe3444e"
server_key_iv: "0341b42a9eb7c472fe65c66fdb980785"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.328644 6971 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:39.331749 6971 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 04:30:39.331841 6971 log.cc:1196] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 04:30:39.332119 6971 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Deleting consensus metadata
I20260620 04:30:39.332276 6971 master_runner.cc:315] Copying system tablet from 127.0.98.62:37507
I20260620 04:30:39.333509 6971 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 04:30:39.336002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39179
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37507,127.0.98.61:45881,127.0.98.60:39179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:30:39.342271 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.334187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.0.98.62:37507)
0620 04:30:39.334318 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.334322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.334374 (+ 52us) server_negotiation.cc:407] Connection header received
0620 04:30:39.334599 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.334603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.334659 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.334774 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.335380 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.335994 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.336978 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.337332 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.337444 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.338711 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.338715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.338718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.338745 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.340419 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.340916 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.340920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.340922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.341031 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.341332 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.341335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.341337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.341559 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.341628 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.341963 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.342090 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:30:39.342836 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.334030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:41810)
0620 04:30:39.334280 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.334294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.334415 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.334889 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.334892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.335136 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.335249 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.335257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.336184 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.336188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.336862 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.336871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.337454 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.337458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.337552 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.338094 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.338109 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.338543 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.340550 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.340553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.340556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.340811 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.341137 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.341140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.341142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.341237 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.341696 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.341699 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.341796 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.342491 (+ 695us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.342647 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260620 04:30:39.343532 6527 tablet_copy_service.cc:140] P 0c6740c79de24b00b6ec13d810e271bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7eec4f1aa71e4e1585702f0c51bb0a9c ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41810)
I20260620 04:30:39.343740 6527 tablet_copy_service.cc:161] P 0c6740c79de24b00b6ec13d810e271bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7eec4f1aa71e4e1585702f0c51bb0a9c at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41810: session id = 7eec4f1aa71e4e1585702f0c51bb0a9c-00000000000000000000000000000000
I20260620 04:30:39.345757 6527 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 04:30:39.347605 6971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 04:30:39.350096 6971 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: tablet copy: Starting download of 0 data blocks...
I20260620 04:30:39.350315 6971 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: tablet copy: Starting download of 1 WAL segments...
I20260620 04:30:39.352195 6971 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 04:30:39.354310 6527 tablet_copy_service.cc:342] P 0c6740c79de24b00b6ec13d810e271bc: Request end of tablet copy session 7eec4f1aa71e4e1585702f0c51bb0a9c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41810
I20260620 04:30:39.354449 6527 tablet_copy_service.cc:433] P 0c6740c79de24b00b6ec13d810e271bc: ending tablet copy session 7eec4f1aa71e4e1585702f0c51bb0a9c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7eec4f1aa71e4e1585702f0c51bb0a9c
I20260620 04:30:39.362524 6971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:39.364362 7080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:39.366098 7081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:39.366322 7083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:39.366381 6971 server_base.cc:1034] running on GCE node
I20260620 04:30:39.366678 6971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:39.366999 6971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:39.368156 6971 hybrid_clock.cc:648] HybridClock initialized: now 1781929839368144 us; error 30 us; skew 500 ppm
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:39.369717 6971 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 04:30:39.370201 6971 webserver.cc:460] Webserver started at http://127.0.98.61:40773/ using document root <none> and password file <none>
I20260620 04:30:39.370388 6971 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:39.370429 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:39.371704 6971 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 04:30:39.372634 7090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.372841 6971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 04:30:39.372895 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal
uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c"
format_stamp: "Formatted at 2026-06-20 04:30:39 on dist-test-slave-rzqm"
server_key: "640e965b7a8ffa458bfca09eefe3444e"
server_key_iv: "0341b42a9eb7c472fe65c66fdb980785"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.372977 6971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:39.380230 6971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:39.382287 6971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:39.386868 6971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45881
I20260620 04:30:39.386897 7142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45881 every 8 connection(s)
I20260620 04:30:39.387430 6971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:30:39.389290 7143 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:30:39.390336 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Bootstrap starting.
I20260620 04:30:39.401042 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:30:39.401400 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Bootstrap complete.
I20260620 04:30:39.402016 7143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } }
I20260620 04:30:39.402192 7143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7eec4f1aa71e4e1585702f0c51bb0a9c, State: Initialized, Role: LEARNER
I20260620 04:30:39.402315 7143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } }
I20260620 04:30:39.402678 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } } }
I20260620 04:30:39.402802 7143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:39.402886 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: This master's current role is: LEARNER
I20260620 04:30:39.406126 7157 catalog_manager.cc:1259] Loaded cluster ID: db6e4e50e6564cd596f8c627a50e4c1b
I20260620 04:30:39.406186 7157 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: loading cluster ID for follower catalog manager: success
I20260620 04:30:39.408164 7157 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: acquiring CA information for follower catalog manager: success
I20260620 04:30:39.409538 7157 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 04:30:39.463939 7079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:39.464238 7079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:39.464290 7079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:39.468386 7079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:39.468451 7079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:39.468470 7079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:39.468485 7079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:39.468499 7079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:39.473886 7079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37507,127.0.98.61:45881,127.0.98.60:39179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:39.474951 7079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:39.475987 7079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:39.484558 7163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:39.484702 7162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:39.484606 7079 server_base.cc:1034] running on GCE node
W20260620 04:30:39.484620 7165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:39.485440 7079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:39.486603 7079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:39.487785 7079 hybrid_clock.cc:648] HybridClock initialized: now 1781929839487757 us; error 42 us; skew 500 ppm
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:39.490509 7079 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 04:30:39.491402 7079 webserver.cc:460] Webserver started at http://127.0.98.60:46135/ using document root <none> and password file <none>
I20260620 04:30:39.491932 7079 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:39.491993 7079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:39.492204 7079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:39.494306 7079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data/instance:
uuid: "f84c158313d640aab3c9cf26b3e54457"
format_stamp: "Formatted at 2026-06-20 04:30:39 on dist-test-slave-rzqm"
server_key: "2f66a8491e10b139bc6081fe973083fc"
server_key_iv: "866265be69ed2b38a467e8afdb73301b"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.494820 7079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal/instance:
uuid: "f84c158313d640aab3c9cf26b3e54457"
format_stamp: "Formatted at 2026-06-20 04:30:39 on dist-test-slave-rzqm"
server_key: "2f66a8491e10b139bc6081fe973083fc"
server_key_iv: "866265be69ed2b38a467e8afdb73301b"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.498646 7079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:39.501276 7172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.502794 7079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:39.502936 7079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal
uuid: "f84c158313d640aab3c9cf26b3e54457"
format_stamp: "Formatted at 2026-06-20 04:30:39 on dist-test-slave-rzqm"
server_key: "2f66a8491e10b139bc6081fe973083fc"
server_key_iv: "866265be69ed2b38a467e8afdb73301b"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.503057 7079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:39.522789 7079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:39.526119 7079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:39.526295 7079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:39.533447 7079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39179
I20260620 04:30:39.533692 7224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39179 every 8 connection(s)
I20260620 04:30:39.534541 7079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:30:39.537595 7225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:39.541674 7225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:30:39.543087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 7079
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:30:39.555400 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.543782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.0.98.62:37507)
0620 04:30:39.543927 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.543931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.544973 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 04:30:39.545890 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.545892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.545942 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.546020 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.547012 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.547573 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.548713 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.548946 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.549068 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.551710 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.551714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.551717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.551745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.553224 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.553802 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.553805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.553806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.553883 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.554215 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.554219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.554221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.554439 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.554533 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.555092 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.555221 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:30:39.556452 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.544243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.0.98.60:39179)
0620 04:30:39.544751 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.544769 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.544812 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:30:39.545640 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.545666 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.545922 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.546087 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.546871 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.547900 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.548751 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.549267 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.549366 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.552099 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.552106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.552122 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.552148 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.553852 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.554230 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.554236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.554238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.554303 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.554565 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.554568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.554569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.554862 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.555031 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.555227 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.555444 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":383,"spinlock_wait_cycles":2432,"thread_start_us":147,"threads_started":1}
I20260620 04:30:39.556452 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.544243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:41816)
0620 04:30:39.544804 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.544859 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.545647 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.546083 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.546089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.546530 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.546867 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.546880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.547693 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.547697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.548602 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.548612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.549108 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.549112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.549316 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.549990 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.550007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.551552 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.553330 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.553337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.553348 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.553680 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.553966 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.553969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.553971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.554122 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.554529 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.554535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.554865 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.555033 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.555383 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":43,"spinlock_wait_cycles":59008,"thread_start_us":157,"threads_started":1}
I20260620 04:30:39.557849 7225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:30:39.566066 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.558432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.0.98.61:45881)
0620 04:30:39.558710 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.558715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.558730 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:39.558772 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.558775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.558822 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.558914 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.559316 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.560074 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.560831 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.561107 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.561190 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.563029 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.563032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.563034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.563060 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.564531 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.564924 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.564928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.564930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.565046 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.565277 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.565281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.565283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.565468 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.565562 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.565803 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.565920 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
I20260620 04:30:39.566061 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.558342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45881 (local address 127.0.0.1:49064)
0620 04:30:39.558453 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.558467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.558540 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.558932 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.558935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.559103 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.559210 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.559215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.560170 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.560172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.560728 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.560735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.561190 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.561193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.561278 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.561693 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.561703 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.562870 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.564637 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.564640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.564642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.564830 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.565124 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.565128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.565130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.565202 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.565562 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.565566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.565656 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.565738 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.565878 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 04:30:39.566937 7225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:30:39.574504 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.567384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:39428)
0620 04:30:39.567493 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.567507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.567577 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.567830 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.567832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.567960 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.568033 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.568038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.568682 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.568684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.569235 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.569243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.569712 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.569714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.569771 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.570178 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.570187 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.571383 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.572953 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.572956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.572959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.573250 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.573521 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.573523 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.573525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.573617 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.574063 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.574066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.574158 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.574250 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.574376 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 04:30:39.574605 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.567512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.0.98.60:39179)
0620 04:30:39.567635 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.567638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.567651 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:39.567679 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.567682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.567743 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.567813 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.568110 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.568610 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.569352 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.569617 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.569699 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.571522 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.571525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.571527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.571548 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.572824 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.573342 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.573345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.573347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.573422 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.573737 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.573740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.573741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.573938 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.574038 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.574331 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.574454 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:30:39.577875 7225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: Bootstrap starting.
I20260620 04:30:39.580215 7225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:39.580962 7225 log.cc:824] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:39.583282 7225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: No bootstrap required, opened a new log
I20260620 04:30:39.587419 7225 raft_consensus.cc:348] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
I20260620 04:30:39.587637 7225 raft_consensus.cc:374] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:39.587678 7225 raft_consensus.cc:729] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84c158313d640aab3c9cf26b3e54457, State: Initialized, Role: FOLLOWER
I20260620 04:30:39.588112 7225 consensus_queue.cc:260] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
I20260620 04:30:39.588549 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } } }
I20260620 04:30:39.588716 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:30:39.589308 7225 sys_catalog.cc:564] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 04:30:39.594013 7242 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:30:39.594105 7242 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:30:39.597893 7079 master_runner.cc:428] Detected that this master f84c158313d640aab3c9cf26b3e54457 is joining an existing cluster
I20260620 04:30:39.597960 7079 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39179
I20260620 04:30:39.606107 6698 heartbeater.cc:503] Master 127.0.98.62:37507 was elected leader, sending a full tablet report...
I20260620 04:30:39.609306 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.601193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.0.98.61:45881)
0620 04:30:39.601310 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.601313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.601326 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:39.601362 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.601364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.601407 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.601475 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.601950 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.602727 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.604072 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.604441 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.604512 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.605877 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.605881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.605884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.605917 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.607561 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.608086 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.608089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.608091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.608183 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.608410 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.608413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.608415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.608628 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.608721 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.608992 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.609148 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:30:39.609321 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.600585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45881 (local address 127.0.0.1:49078)
0620 04:30:39.600838 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.600850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.600928 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.601573 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.601576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.601733 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.601824 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.601829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.602860 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.602863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.603935 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.603946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.604693 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.604696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.604832 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.605347 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.605362 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.605740 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.607681 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.607685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.607687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.607931 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.608260 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.608263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.608264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.608338 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.608738 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.608740 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.608844 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.608938 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.609113 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":117,"thread_start_us":88,"threads_started":1}
I20260620 04:30:39.610252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 6569
I20260620 04:30:39.616605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45291
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37507
--webserver_port=46753
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37507,127.0.98.61:45881
--tserver_master_addrs=127.0.98.62:37507,127.0.98.61:45881,127.0.98.60:39179 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:30:39.629822 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.600232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.0.98.60:39179)
0620 04:30:39.600967 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.600971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.600983 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:30:39.601034 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.601037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.601074 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.601137 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.601701 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.602429 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.604137 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.604445 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.604511 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.626509 (+ 21998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.626514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.626517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.626541 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.628102 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.628579 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.628582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.628584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.628659 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.628934 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.628936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.628938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.629192 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.629259 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.629521 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.629637 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678}
I20260620 04:30:39.630086 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.600845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.0.98.62:37507)
0620 04:30:39.600989 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.600993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.601039 (+ 46us) server_negotiation.cc:407] Connection header received
0620 04:30:39.601080 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.601083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.601130 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.601203 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.601807 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.602608 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.604163 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.604500 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.604572 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.626540 (+ 21968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.626545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.626548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.626580 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.628283 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.628754 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.628758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.628760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.628847 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.629197 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.629200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.629202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.629416 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.629482 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.629812 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.629931 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:30:39.630642 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.600149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:41826)
0620 04:30:39.600421 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.600437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.600531 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.601318 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.601321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.601539 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.601656 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.601663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.602763 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.602766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.603993 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.604004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.604720 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.604723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.604910 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.625660 (+ 20750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.625678 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.626309 (+ 631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.628416 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.628419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.628422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.628647 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.628947 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.628950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.628952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.629084 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.629547 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.629550 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.629644 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.630291 (+ 647us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.630446 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":121,"spinlock_wait_cycles":8320,"thread_start_us":113,"threads_started":1}
I20260620 04:30:39.631042 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.600250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:39434)
0620 04:30:39.600567 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.600581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.600664 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.601238 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.601241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.601472 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.601584 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.601591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.602537 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.602540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.604009 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.604017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.604693 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.604696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.604868 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.625595 (+ 20727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.625615 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.626309 (+ 694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.628239 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.628242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.628245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.628476 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.628767 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.628770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.628772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.628851 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.629310 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.629312 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.629383 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.630389 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.630529 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":144,"spinlock_wait_cycles":15104,"thread_start_us":93,"threads_started":1}
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929838, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:30:39.644587 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.634697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:39448)
0620 04:30:39.634951 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.634966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.635044 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.635686 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.635690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.635909 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.636028 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.636036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.637127 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.637131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.638028 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.638041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.638736 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.638741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.638845 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.639415 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.639428 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.641078 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.642989 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.642992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.642994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.643237 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.643528 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.643531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.643533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.643615 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.644058 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.644061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.644162 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.644257 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.644417 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260620 04:30:39.645445 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.635187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.0.98.60:39179)
0620 04:30:39.635306 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.635310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.635322 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:30:39.635401 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.635404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.635460 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.635539 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.636158 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.636953 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.638177 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.638493 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.638585 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.641241 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.641245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.641249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.641277 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.642858 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.643347 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.643351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.643353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.643439 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.643700 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.643703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.643705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.643928 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.644014 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.645164 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.645288 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:30:39.646445 6507 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39179: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f84c158313d640aab3c9cf26b3e54457"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39179
}
attrs {
promote: true
}
}
dest_uuid: "0c6740c79de24b00b6ec13d810e271bc"
cas_config_opid_index: 5
I20260620 04:30:39.647042 6507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } }
I20260620 04:30:39.648379 7066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f84c158313d640aab3c9cf26b3e54457 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.649863 7066 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39179
I20260620 04:30:39.650044 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.650256 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:39.650624 7079 master.cc:537] Master@127.0.98.60:39179 shutting down...
I20260620 04:30:39.651036 7199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:39.652482 7199 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Refusing update from remote peer 0c6740c79de24b00b6ec13d810e271bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 04:30:39.655022 7252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:39.656432 7231 raft_consensus.cc:482] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:39.656596 7231 raft_consensus.cc:504] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } }
W20260620 04:30:39.656960 6492 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc -> Peer f84c158313d640aab3c9cf26b3e54457 (127.0.98.60:39179): Couldn't send request to peer f84c158313d640aab3c9cf26b3e54457. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 04:30:39.657704 7231 leader_election.cc:288] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507), 7eec4f1aa71e4e1585702f0c51bb0a9c (127.0.98.61:45881)
I20260620 04:30:39.658236 6517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f84c158313d640aab3c9cf26b3e54457" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c6740c79de24b00b6ec13d810e271bc" is_pre_election: true
I20260620 04:30:39.658846 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.648595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45881 (local address 127.0.0.1:49082)
0620 04:30:39.653351 (+ 4756us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.653366 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.653464 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.653870 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.653873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.653890 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:39.653981 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.653986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.655006 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.655010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.656901 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.656912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.658301 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.658305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.658486 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.658552 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.658623 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.658701 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4679}
I20260620 04:30:39.658795 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f84c158313d640aab3c9cf26b3e54457" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" is_pre_election: true
I20260620 04:30:39.658951 7117 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 LEARNER]: Handling vote request from an unknown peer f84c158313d640aab3c9cf26b3e54457
I20260620 04:30:39.659068 7117 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate f84c158313d640aab3c9cf26b3e54457 for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 04:30:39.659350 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.648869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.0.98.61:45881)
0620 04:30:39.653248 (+ 4379us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.653254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.653547 (+ 293us) server_negotiation.cc:407] Connection header received
0620 04:30:39.653602 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.653605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.653664 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.653746 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:39.654090 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.654877 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.657244 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.657970 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.658071 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.658240 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.659157 (+ 917us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.659241 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4277}
I20260620 04:30:39.659727 7173 leader_election.cc:302] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f84c158313d640aab3c9cf26b3e54457; no voters: 0c6740c79de24b00b6ec13d810e271bc, 7eec4f1aa71e4e1585702f0c51bb0a9c
I20260620 04:30:39.660137 7231 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:30:39.660265 7117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 LEARNER]: Refusing update from remote peer 0c6740c79de24b00b6ec13d810e271bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 04:30:39.660816 7252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 04:30:39.660948 7079 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:39.661218 7079 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:39.661311 7079 tablet_replica.cc:330] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: stopping tablet replica
I20260620 04:30:39.663044 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f84c158313d640aab3c9cf26b3e54457 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.666555 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c6740c79de24b00b6ec13d810e271bc. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.666679 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: This master's current role is: LEARNER
I20260620 04:30:39.666771 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: true } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.666832 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: This master's current role is: LEARNER
I20260620 04:30:39.667651 7252 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: attempting to promote NON_VOTER 7eec4f1aa71e4e1585702f0c51bb0a9c to VOTER
I20260620 04:30:39.668183 7252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } }
I20260620 04:30:39.669466 7117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 LEARNER]: Refusing update from remote peer 0c6740c79de24b00b6ec13d810e271bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 04:30:39.669888 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 04:30:39.670137 6492 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc -> Peer f84c158313d640aab3c9cf26b3e54457 (127.0.98.60:39179): Couldn't send request to peer f84c158313d640aab3c9cf26b3e54457. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 04:30:39.671775 7067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7eec4f1aa71e4e1585702f0c51bb0a9c (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.673545 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.673657 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:39.673974 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.674089 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:39.674463 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7eec4f1aa71e4e1585702f0c51bb0a9c (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.675928 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.676090 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:30:39.683708 7079 master.cc:552] Master@127.0.98.60:39179 shutdown complete.
I20260620 04:30:39.683779 7079 master_runner.cc:305] Clearing existing system tablet
I20260620 04:30:39.684012 7079 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:39.684069 7079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:39.685417 7079 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 04:30:39.686291 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.686542 7079 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:39.686666 7079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:39.686774 7079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal
uuid: "f84c158313d640aab3c9cf26b3e54457"
format_stamp: "Formatted at 2026-06-20 04:30:39 on dist-test-slave-rzqm"
server_key: "2f66a8491e10b139bc6081fe973083fc"
server_key_iv: "866265be69ed2b38a467e8afdb73301b"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.688815 7079 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:39.691202 7079 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 04:30:39.691294 7079 log.cc:1196] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 04:30:39.691515 7079 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: Deleting consensus metadata
I20260620 04:30:39.691668 7079 master_runner.cc:315] Copying system tablet from 127.0.98.62:37507
I20260620 04:30:39.692821 7079 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37507
I20260620 04:30:39.706178 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.693384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:41836)
0620 04:30:39.697841 (+ 4457us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.697859 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.698076 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.698395 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.698399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.698613 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.698754 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.698762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.699856 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.699860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.700722 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.700734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.701340 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.701344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.701428 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.701972 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.701986 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.702430 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.704424 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.704428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.704430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.704670 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.705095 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.705099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.705101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.705194 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.705614 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.705617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.705731 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.705835 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.705986 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4367,"thread_start_us":87,"threads_started":1}
I20260620 04:30:39.706195 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.693478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.0.98.62:37507)
0620 04:30:39.693604 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.693609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.697983 (+ 4374us) server_negotiation.cc:407] Connection header received
0620 04:30:39.698204 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.698209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.698269 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.698370 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.698890 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.699738 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.700852 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.701224 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.701328 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.702586 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.702590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.702593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.702622 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.704314 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.704856 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.704859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.704861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.704952 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.705279 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.705281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.705283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.705525 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.705622 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.705894 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.706019 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:30:39.706876 6527 tablet_copy_service.cc:140] P 0c6740c79de24b00b6ec13d810e271bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f84c158313d640aab3c9cf26b3e54457 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41836)
I20260620 04:30:39.706975 6527 tablet_copy_service.cc:161] P 0c6740c79de24b00b6ec13d810e271bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f84c158313d640aab3c9cf26b3e54457 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41836: session id = f84c158313d640aab3c9cf26b3e54457-00000000000000000000000000000000
I20260620 04:30:39.708005 6527 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 04:30:39.709270 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 04:30:39.711702 7079 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: tablet copy: Starting download of 0 data blocks...
I20260620 04:30:39.711907 7079 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: tablet copy: Starting download of 1 WAL segments...
I20260620 04:30:39.713655 7079 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 04:30:39.715742 6527 tablet_copy_service.cc:342] P 0c6740c79de24b00b6ec13d810e271bc: Request end of tablet copy session f84c158313d640aab3c9cf26b3e54457-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41836
I20260620 04:30:39.715860 6527 tablet_copy_service.cc:433] P 0c6740c79de24b00b6ec13d810e271bc: ending tablet copy session f84c158313d640aab3c9cf26b3e54457-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f84c158313d640aab3c9cf26b3e54457
I20260620 04:30:39.723551 7079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:39.725523 7267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:39.725826 7264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:39.729190 7265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:39.730159 7079 server_base.cc:1034] running on GCE node
I20260620 04:30:39.730348 7079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:39.730711 7079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:39.731868 7079 hybrid_clock.cc:648] HybridClock initialized: now 1781929839731855 us; error 35 us; skew 500 ppm
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:39.733582 7079 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 04:30:39.734130 7079 webserver.cc:460] Webserver started at http://127.0.98.60:33149/ using document root <none> and password file <none>
I20260620 04:30:39.734328 7079 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:39.734370 7079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:39.735673 7079 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:39.736631 7274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.736896 7079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:39.736970 7079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal
uuid: "f84c158313d640aab3c9cf26b3e54457"
format_stamp: "Formatted at 2026-06-20 04:30:39 on dist-test-slave-rzqm"
server_key: "2f66a8491e10b139bc6081fe973083fc"
server_key_iv: "866265be69ed2b38a467e8afdb73301b"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.737077 7079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:39.752094 7079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:39.754166 7079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:39.759323 7079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39179
I20260620 04:30:39.760094 7079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:30:39.761109 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39179 every 8 connection(s)
I20260620 04:30:39.762228 7330 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:30:39.763337 7330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: Bootstrap starting.
W20260620 04:30:39.765928 7251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:39.766263 7251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:39.766345 7251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:39.771881 7251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:39.771975 7251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:39.772065 7251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:39.774539 7330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:30:39.774881 7330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: Bootstrap complete.
I20260620 04:30:39.775532 7330 raft_consensus.cc:348] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } }
I20260620 04:30:39.775683 7330 raft_consensus.cc:729] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f84c158313d640aab3c9cf26b3e54457, State: Initialized, Role: LEARNER
I20260620 04:30:39.775831 7330 consensus_queue.cc:260] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } }
I20260620 04:30:39.776181 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:39.776293 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:30:39.776296 7330 sys_catalog.cc:564] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:39.779196 7344 catalog_manager.cc:1259] Loaded cluster ID: db6e4e50e6564cd596f8c627a50e4c1b
I20260620 04:30:39.779301 7344 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: loading cluster ID for follower catalog manager: success
I20260620 04:30:39.779649 7251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46753
--tserver_master_addrs=127.0.98.62:37507,127.0.98.61:45881,127.0.98.60:39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:39.780731 7251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:39.780812 7344 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: acquiring CA information for follower catalog manager: success
I20260620 04:30:39.781765 7251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:30:39.781934 7344 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 04:30:39.791141 7346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:39.791371 7349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:39.791684 7251 server_base.cc:1034] running on GCE node
W20260620 04:30:39.792460 7347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:39.792896 7251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:39.793571 7251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:39.794741 7251 hybrid_clock.cc:648] HybridClock initialized: now 1781929839794720 us; error 34 us; skew 500 ppm
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:39.797499 7251 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:30:39.798337 7251 webserver.cc:460] Webserver started at http://127.0.98.1:46753/ using document root <none> and password file <none>
I20260620 04:30:39.798863 7251 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:39.798936 7251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:39.802709 7251 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:30:39.804808 7356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.806063 7251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:39.806165 7251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "bac4c91bd5224017a643817100960103"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "c64f53b7afaf05c8c40be11df6a4bd8c"
server_key_iv: "10ca286827b247841be430a0952b6d5c"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.806711 7251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:39.810178 6832 heartbeater.cc:503] Master 127.0.98.62:37507 was elected leader, sending a full tablet report...
I20260620 04:30:39.817334 7251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:39.820399 7251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:39.820574 7251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:39.821221 7251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:39.822093 7251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:39.822139 7251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.822181 7251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:39.822209 7251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.831341 7251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45291
I20260620 04:30:39.831369 7469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45291 every 8 connection(s)
I20260620 04:30:39.832343 7251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:39.836876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 7251
I20260620 04:30:39.837111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 6703
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:30:39.843871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35869
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37507
--webserver_port=35219
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37507,127.0.98.61:45881
--tserver_master_addrs=127.0.98.62:37507,127.0.98.61:45881,127.0.98.60:39179 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:30:39.848927 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.835984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38343 (local address 127.0.98.61:45881)
0620 04:30:39.836114 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.836118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.836140 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:30:39.836399 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.836402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.836457 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.836526 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.838244 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.839003 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.842024 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.842323 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.842404 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.845026 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.845030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.845033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.845062 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.846578 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.847151 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.847155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.847157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.847237 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.847608 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.847611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.847613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.847846 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.847918 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.848639 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.848764 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:30:39.849174 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.834158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47179 (local address 127.0.98.60:39179)
0620 04:30:39.834381 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.834391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.835661 (+ 1270us) server_negotiation.cc:407] Connection header received
0620 04:30:39.836443 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.836447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.836499 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.836602 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.838236 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.839293 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.841251 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.841586 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.841656 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.844835 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.844839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.844842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.844869 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.846468 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.847147 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.847150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.847152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.847266 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.847624 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.847627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.847629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.847845 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.847932 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.848840 (+ 908us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.848974 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260620 04:30:39.850046 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.835766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45881 (local address 127.0.98.1:38343)
0620 04:30:39.835981 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.835993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.836217 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.836759 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.836768 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.837735 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.838083 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.838102 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.839176 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.839179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.841150 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.841156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.842565 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.842568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.842658 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.843367 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.843378 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.844844 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.846685 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.846688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.846690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.847047 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.847338 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.847341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.847343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.847512 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.847980 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.847987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.848351 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.849102 (+ 751us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.849233 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":643,"spinlock_wait_cycles":24064,"thread_start_us":67,"threads_started":1}
I20260620 04:30:39.850311 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.834671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.1:36809)
0620 04:30:39.835200 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.835242 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.836213 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.836757 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.836767 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.838424 (+ 1657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.838534 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.838541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.839564 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.839567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.840756 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.840766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.841481 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.841485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.841670 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.842305 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.842326 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.844303 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.847003 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.847006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.847007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.847238 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.847509 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.847512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.847514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.847587 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.847980 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.847987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.848351 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.848564 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.848918 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":169,"spinlock_wait_cycles":33408,"thread_start_us":652,"threads_started":1}
I20260620 04:30:39.850994 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.834671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.98.1:47179)
0620 04:30:39.835567 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:39.835579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:39.836216 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:39.836757 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:39.836767 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:39.837405 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:39.838086 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.838103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.839420 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.839423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.840975 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:39.840982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.841798 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.841801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:39.841881 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.842894 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:39.842908 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:39.844690 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:39.846586 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.846592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.846611 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.847042 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.847358 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:39.847361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:39.847363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:39.847521 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:39.847982 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:39.847988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:39.848353 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 04:30:39.848564 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 04:30:39.850837 (+ 2273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":2515,"spinlock_wait_cycles":47360,"thread_start_us":138,"threads_started":1}
I20260620 04:30:39.851552 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:39.834158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36809 (local address 127.0.98.62:37507)
0620 04:30:39.834288 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:39.834292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:39.835356 (+ 1064us) server_negotiation.cc:407] Connection header received
0620 04:30:39.836443 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:39.836447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:39.836499 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:39.836553 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:39.838666 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.839449 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.840879 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:39.841226 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:39.841320 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:39.844490 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:39.844494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:39.844497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:39.844527 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:39.846889 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.847339 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.847342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.847344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.847416 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:39.847675 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:39.847678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:39.847680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:39.847856 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:39.847930 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:39.851284 (+ 3354us) server_negotiation.cc:299] Negotiation successful
0620 04:30:39.851416 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:30:39.851563 7472 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45881
I20260620 04:30:39.851760 7472 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:39.852011 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39179
I20260620 04:30:39.852146 7470 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:39.852389 7470 heartbeater.cc:511] Master 127.0.98.60:39179 requested a full tablet report, sending...
I20260620 04:30:39.852389 7472 heartbeater.cc:511] Master 127.0.98.61:45881 requested a full tablet report, sending...
I20260620 04:30:39.852667 7471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37507
I20260620 04:30:39.852737 7471 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:39.852886 7471 heartbeater.cc:511] Master 127.0.98.62:37507 requested a full tablet report, sending...
I20260620 04:30:39.853914 7291 ts_manager.cc:194] Registered new tserver with Master: bac4c91bd5224017a643817100960103 (127.0.98.1:45291)
I20260620 04:30:39.854013 7099 ts_manager.cc:194] Registered new tserver with Master: bac4c91bd5224017a643817100960103 (127.0.98.1:45291)
I20260620 04:30:39.854789 6507 ts_manager.cc:194] Re-registered known tserver with Master: bac4c91bd5224017a643817100960103 (127.0.98.1:45291)
I20260620 04:30:39.855590 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36809
W20260620 04:30:39.959743 7478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:39.960018 7478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:39.960060 7478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:39.964174 7478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:39.964251 7478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:39.964310 7478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:39.969875 7478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35219
--tserver_master_addrs=127.0.98.62:37507,127.0.98.61:45881,127.0.98.60:39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:39.971035 7478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:39.972085 7478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:39.981894 7483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:39.981907 7484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:39.981986 7478 server_base.cc:1034] running on GCE node
W20260620 04:30:39.981907 7486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:39.982635 7478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:39.983256 7478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:39.984404 7478 hybrid_clock.cc:648] HybridClock initialized: now 1781929839984386 us; error 38 us; skew 500 ppm
Jun 20 04:30:39 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:39.987130 7478 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:30:39.988001 7478 webserver.cc:460] Webserver started at http://127.0.98.2:35219/ using document root <none> and password file <none>
I20260620 04:30:39.988529 7478 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:39.988610 7478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:39.992508 7478 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:39.994663 7493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:39.995962 7478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:39.996060 7478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "50af5eaac37b4e4690ea841a0a67096a"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "24f871ce64617dd7e04041134981aac4"
server_key_iv: "a23cbc724901e61c3e7e4b032e8089f3"
server_key_version: "encryptionkey@0"
I20260620 04:30:39.996613 7478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:40.022768 7478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:40.023648 6966 heartbeater.cc:503] Master 127.0.98.62:37507 was elected leader, sending a full tablet report...
I20260620 04:30:40.026453 7478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:40.026614 7478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:40.027171 7478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:40.028002 7478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:40.028049 7478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:40.028090 7478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:40.028103 7478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:40.036429 7478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35869
I20260620 04:30:40.036463 7606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35869 every 8 connection(s)
I20260620 04:30:40.037488 7478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:40.041312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 7478
I20260620 04:30:40.041505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 6837
I20260620 04:30:40.048062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42819
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37507
--webserver_port=40785
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37507,127.0.98.61:45881
--tserver_master_addrs=127.0.98.62:37507,127.0.98.61:45881,127.0.98.60:39179 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:30:40 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 04:30:40 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 04:30:40 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 04:30:40.056013 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.041102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52151 (local address 127.0.98.62:37507)
0620 04:30:40.041223 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.041227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.041340 (+ 113us) server_negotiation.cc:407] Connection header received
0620 04:30:40.043150 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.043154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.043211 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.043264 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.046082 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.046844 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.048019 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.048310 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.048381 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.051582 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.051585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.051588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.051613 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.053499 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.054204 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.054208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.054209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.054290 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.054691 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.054694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.054695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.054897 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.054975 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.055727 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.055849 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:30:40.056013 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.039275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55315 (local address 127.0.98.61:45881)
0620 04:30:40.039402 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.039406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.040807 (+ 1401us) server_negotiation.cc:407] Connection header received
0620 04:30:40.043234 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.043238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.043289 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.043355 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.045207 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.045813 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.047543 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.047882 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.047983 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.051442 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.051446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.051449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.051477 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.053499 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.054205 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.054208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.054209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.054290 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.054678 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.054681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.054683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.054907 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.054981 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.055727 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.055854 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:30:40.056476 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.039289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39981 (local address 127.0.98.60:39179)
0620 04:30:40.039402 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.039406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.040502 (+ 1096us) server_negotiation.cc:407] Connection header received
0620 04:30:40.043145 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.043149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.043212 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.043263 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.046205 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.046923 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.048072 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.048381 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.048453 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.051758 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.051762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.051765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.051795 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.053443 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.054245 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.054249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.054251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.054339 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.054696 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.054699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.054701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.054914 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.055007 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.056187 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.056319 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:30:40.057783 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.039822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45881 (local address 127.0.98.2:55315)
0620 04:30:40.040719 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.040731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.042875 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.043606 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.043616 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.044208 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:40.045031 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.045053 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.045957 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.045960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.047412 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.047421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.048167 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.048171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.048363 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.049306 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:40.049320 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:40.051254 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:40.053619 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.053623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.053624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.054097 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.054383 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.054386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.054388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.054574 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.055046 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:40.055053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:40.055427 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.056734 (+ 1307us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.056887 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":229,"spinlock_wait_cycles":68224,"thread_start_us":140,"threads_started":1}
I20260620 04:30:40.058040 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.040957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.2:52151)
0620 04:30:40.041247 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.041270 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.042885 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.043606 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.043616 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.044206 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:40.045942 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.045953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.046977 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.046979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.047904 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.047912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.048520 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.048524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.048620 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.049065 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:40.049087 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:40.051423 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:40.053641 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.053643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.053645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.054094 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.054385 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.054388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.054390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.054577 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.055061 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:40.055063 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:40.055427 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.056248 (+ 821us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.056622 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":1072,"spinlock_wait_cycles":43520,"thread_start_us":106,"threads_started":1}
I20260620 04:30:40.057789 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.039822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.98.2:39981)
0620 04:30:40.040349 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.040393 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.042875 (+ 2482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.043605 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.043616 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.044385 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:40.046053 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.046063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.047078 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.047081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.047935 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.047945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.048612 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.048615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.048697 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.049196 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:40.049210 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:40.051616 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:40.053577 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.053584 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.053602 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.054127 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.054447 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.054450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.054451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.054586 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.055065 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:40.055067 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:40.055430 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.056673 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.056828 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":1355,"spinlock_wait_cycles":3072,"thread_start_us":651,"threads_started":1}
I20260620 04:30:40.059237 7608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37507
I20260620 04:30:40.059310 7609 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45881
I20260620 04:30:40.059423 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39179
I20260620 04:30:40.059509 7609 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:40.059508 7607 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:40.059509 7608 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:40.060154 7607 heartbeater.cc:511] Master 127.0.98.60:39179 requested a full tablet report, sending...
I20260620 04:30:40.060148 7608 heartbeater.cc:511] Master 127.0.98.62:37507 requested a full tablet report, sending...
I20260620 04:30:40.060149 7609 heartbeater.cc:511] Master 127.0.98.61:45881 requested a full tablet report, sending...
I20260620 04:30:40.061264 6507 ts_manager.cc:194] Re-registered known tserver with Master: 50af5eaac37b4e4690ea841a0a67096a (127.0.98.2:35869)
I20260620 04:30:40.061314 7291 ts_manager.cc:194] Registered new tserver with Master: 50af5eaac37b4e4690ea841a0a67096a (127.0.98.2:35869)
I20260620 04:30:40.062016 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52151
I20260620 04:30:40.062538 7099 ts_manager.cc:194] Registered new tserver with Master: 50af5eaac37b4e4690ea841a0a67096a (127.0.98.2:35869)
W20260620 04:30:40.163753 7614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:40.164059 7614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:40.164105 7614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:40.168443 7614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:40.168525 7614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:40.168591 7614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:40.174412 7614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40785
--tserver_master_addrs=127.0.98.62:37507,127.0.98.61:45881,127.0.98.60:39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:40.175559 7614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:40.176726 7614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:40.186597 7620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:40.186615 7619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:40.186615 7622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:40.186837 7614 server_base.cc:1034] running on GCE node
I20260620 04:30:40.187245 7614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:40.187937 7614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:40.189088 7614 hybrid_clock.cc:648] HybridClock initialized: now 1781929840189057 us; error 43 us; skew 500 ppm
Jun 20 04:30:40 dist-test-slave-rzqm krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:40.191869 7614 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:30:40.192802 7614 webserver.cc:460] Webserver started at http://127.0.98.3:40785/ using document root <none> and password file <none>
I20260620 04:30:40.193444 7614 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:40.193529 7614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:40.197468 7614 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:30:40.199688 7629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:40.201047 7614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:40.201165 7614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "2919829f57ac4813af09f80a642e8f97"
format_stamp: "Formatted at 2026-06-20 04:30:38 on dist-test-slave-rzqm"
server_key: "4bff71da1798846f8423844e6b69369c"
server_key_iv: "43328aeb8136fd5570bd4644182468e0"
server_key_version: "encryptionkey@0"
I20260620 04:30:40.201719 7614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:40.247954 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.244588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.0.98.60:39179)
0620 04:30:40.244715 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.244720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.244827 (+ 107us) server_negotiation.cc:407] Connection header received
0620 04:30:40.244973 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.244975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.245069 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.245182 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:40.245457 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.245979 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.247035 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.247451 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.247541 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.247714 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.247749 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.247823 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:30:40.247956 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.244473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:37746)
0620 04:30:40.244715 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.244731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.244860 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.245211 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.245213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.245248 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:40.245352 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.245358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.246092 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.246095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.246909 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.246919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.247560 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.247564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.247649 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.247673 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.247733 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.247812 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260620 04:30:40.248716 7296 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 04:30:40.249176 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c6740c79de24b00b6ec13d810e271bc. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:40.249325 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: This master's current role is: LEARNER
I20260620 04:30:40.251153 7614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:40.254393 7614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:40.254570 7614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:40.255195 7614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:40.256064 7614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:40.256110 7614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:40.256152 7614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:40.256181 7614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:40.264694 7614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42819
I20260620 04:30:40.264741 7744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42819 every 8 connection(s)
I20260620 04:30:40.265820 7614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:40.275787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 7614
Jun 20 04:30:40 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 04:30:40 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 04:30:40 dist-test-slave-rzqm krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 04:30:40.282688 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.267631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55035 (local address 127.0.98.61:45881)
0620 04:30:40.267762 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.267766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.268948 (+ 1182us) server_negotiation.cc:407] Connection header received
0620 04:30:40.270115 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.270118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.270171 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.270222 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.271836 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.272376 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.273969 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.274276 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.274372 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.278867 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.278871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.278874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.278902 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.280701 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.281176 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.281179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.281181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.281276 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.281542 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.281545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.281547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.281740 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.281823 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.282409 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.282527 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:30:40.283134 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.268197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.98.3:55283)
0620 04:30:40.269124 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.269136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.269881 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.270444 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.270451 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.270949 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:40.271654 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.271666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.272751 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.272754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.273867 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.273873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.274570 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.274573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.274654 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.275391 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:40.275403 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:40.277341 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:40.279733 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.279740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.279759 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.280041 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.280370 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.280373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.280375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.280564 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.281058 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:40.281066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:40.281420 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.281636 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.282008 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":142,"spinlock_wait_cycles":94464,"thread_start_us":146,"threads_started":1}
I20260620 04:30:40.283134 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.269392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.3:49803)
0620 04:30:40.269637 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.269650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.269886 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.270443 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.270450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.270941 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:40.271628 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.271646 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.273108 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.273112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.273869 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.273880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.274540 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.274544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.274632 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.275289 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:40.275304 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:40.278915 (+ 3611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:40.281051 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.281053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.281055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.281266 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.281553 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.281556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.281558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.281636 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.282053 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:40.282056 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:40.282157 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.282237 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.282358 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":75,"spinlock_wait_cycles":18048,"thread_start_us":77,"threads_started":1}
I20260620 04:30:40.283823 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.268197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45881 (local address 127.0.98.3:55035)
0620 04:30:40.268760 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.268828 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.269881 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.270444 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.270451 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.270948 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:40.271676 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.271683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.272533 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.272536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.273853 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.273864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.274421 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.274425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.274593 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.275277 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:40.275294 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:40.278726 (+ 3432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:40.280811 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.280814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.280816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.281073 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.281366 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:40.281369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:40.281371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:40.281452 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:40.281864 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:40.281867 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:40.281951 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.282060 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.282189 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":187,"spinlock_wait_cycles":12416,"thread_start_us":678,"threads_started":1}
I20260620 04:30:40.283986 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.269514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49803 (local address 127.0.98.62:37507)
0620 04:30:40.269641 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.269645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.269719 (+ 74us) server_negotiation.cc:407] Connection header received
0620 04:30:40.270092 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.270095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.270151 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.270222 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.271801 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.272614 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.274020 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.274345 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.274431 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.279082 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.279087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.279090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.279122 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.280928 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.281372 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.281375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.281377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.281459 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.281723 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.281726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.281728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.281927 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.282013 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.283248 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.283851 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:30:40.284246 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.267631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55283 (local address 127.0.98.60:39179)
0620 04:30:40.267838 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.267841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.269210 (+ 1369us) server_negotiation.cc:407] Connection header received
0620 04:30:40.270113 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.270117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.270173 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.270221 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.271822 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.272613 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.273989 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.274330 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.274407 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.277526 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.277530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.277533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.277561 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.279592 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.280156 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.280160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.280162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.280250 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.280672 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.280675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.280678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.280897 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.280986 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.283994 (+ 3008us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.284113 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260620 04:30:40.285373 7745 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39179
I20260620 04:30:40.285399 7747 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45881
I20260620 04:30:40.285445 7746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37507
I20260620 04:30:40.285626 7747 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:40.285624 7745 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:40.286196 7747 heartbeater.cc:511] Master 127.0.98.61:45881 requested a full tablet report, sending...
I20260620 04:30:40.286194 7745 heartbeater.cc:511] Master 127.0.98.60:39179 requested a full tablet report, sending...
I20260620 04:30:40.285624 7746 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:40.286478 7746 heartbeater.cc:511] Master 127.0.98.62:37507 requested a full tablet report, sending...
I20260620 04:30:40.287405 7291 ts_manager.cc:194] Registered new tserver with Master: 2919829f57ac4813af09f80a642e8f97 (127.0.98.3:42819)
I20260620 04:30:40.287405 7099 ts_manager.cc:194] Registered new tserver with Master: 2919829f57ac4813af09f80a642e8f97 (127.0.98.3:42819)
I20260620 04:30:40.287907 6507 ts_manager.cc:194] Re-registered known tserver with Master: 2919829f57ac4813af09f80a642e8f97 (127.0.98.3:42819)
I20260620 04:30:40.288566 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49803
I20260620 04:30:40.291468 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.277878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.0.98.62:37507)
0620 04:30:40.279532 (+ 1654us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.279537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.279550 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:40.279591 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.279594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.279638 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.279721 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.280591 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.281406 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.282525 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.282889 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.282978 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.284527 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.284531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.284534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.284557 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.287648 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.288175 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.288179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.288181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.288259 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.288507 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.288510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.288512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.288927 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.291103 (+ 2176us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.291175 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.291306 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1577,"thread_start_us":65,"threads_started":1}
I20260620 04:30:40.293857 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.278337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.0.98.61:45881)
0620 04:30:40.278942 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.278946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.285031 (+ 6085us) server_negotiation.cc:407] Connection header received
0620 04:30:40.285193 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.285196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.285274 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.285331 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.286156 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.286892 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.287914 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.288199 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.288283 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.290040 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.290044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.290047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.290072 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.291609 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.292525 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.292528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.292529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.292604 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.292887 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.292891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.292893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.293138 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.293238 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.293552 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.293671 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":95,"threads_started":1}
I20260620 04:30:40.294148 6507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:30:40.296370 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.284682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.0.98.60:39179)
0620 04:30:40.288985 (+ 4303us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.288989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.289027 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:30:40.289059 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.289061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.289103 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.289174 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.289943 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.290630 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.291626 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.291898 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.291956 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.293411 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.293415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.293417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.293441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.294864 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.295298 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.295302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.295303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.295376 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.295614 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.295618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.295620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.295787 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.295899 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.296107 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.296211 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4227}
W20260620 04:30:40.297351 6507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:40.299850 7634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: true } } }
I20260620 04:30:40.299984 7634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:40.303571 7762 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: attempting to promote NON_VOTER f84c158313d640aab3c9cf26b3e54457 to VOTER
I20260620 04:30:40.303962 7762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } }
I20260620 04:30:40.305141 7117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Refusing update from remote peer 0c6740c79de24b00b6ec13d810e271bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 04:30:40.305564 7763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 04:30:40.305655 7296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 LEARNER]: Refusing update from remote peer 0c6740c79de24b00b6ec13d810e271bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 04:30:40.306064 7763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 04:30:40.307762 7634 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f84c158313d640aab3c9cf26b3e54457 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:40.308425 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f84c158313d640aab3c9cf26b3e54457 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:40.309262 7634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:40.309360 7634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:40.309478 7634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:40.309540 7634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:40.310253 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:40.310350 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:40.311717 7771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:40.312033 7771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:30:40.312618 7296 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f84c158313d640aab3c9cf26b3e54457 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:40.314324 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c6740c79de24b00b6ec13d810e271bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:40.314481 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:30:40.324580 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.319279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42819 (local address 127.0.0.1:43264)
0620 04:30:40.319560 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.319572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.319644 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.320409 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.320412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.320444 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:40.320696 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.320702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.321724 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.321727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.322939 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.322948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.324117 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.324121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.324214 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.324268 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.324342 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.324412 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":3,"spinlock_wait_cycles":384,"thread_start_us":108,"threads_started":1}
I20260620 04:30:40.324636 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.319698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.0.98.3:42819)
0620 04:30:40.320003 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.320014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.320031 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:30:40.320091 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.320095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.320247 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.320412 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:40.320794 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.321616 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.323089 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.323933 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.324087 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.324240 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.324385 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.324485 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":112,"threads_started":1}
I20260620 04:30:40.324618 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.319039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35869 (local address 127.0.0.1:54658)
0620 04:30:40.319290 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.319305 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.319391 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.320405 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.320408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.320425 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:40.320552 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.320560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.321885 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.321887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.322962 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.322968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.324260 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.324263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.324353 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.324374 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.324435 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.324502 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":15,"spinlock_wait_cycles":768}
I20260620 04:30:40.324939 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.319298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.0.98.2:35869)
0620 04:30:40.319885 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.319897 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.319921 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:30:40.320006 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.320012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.320247 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.320553 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:40.321023 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.321787 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.323089 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.323933 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.324042 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.324201 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.324717 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.324832 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":104,"threads_started":1}
I20260620 04:30:40.328091 7541 tablet_service.cc:1467] Processing CreateTablet for tablet f6161a7e51094f7993bedc7e98f68c27 (DEFAULT_TABLE table=test-table [id=ec3ed84f9b31464e88a9e9f3cbbec566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:40.328091 7679 tablet_service.cc:1467] Processing CreateTablet for tablet f6161a7e51094f7993bedc7e98f68c27 (DEFAULT_TABLE table=test-table [id=ec3ed84f9b31464e88a9e9f3cbbec566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:40.329293 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6161a7e51094f7993bedc7e98f68c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:40.329291 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6161a7e51094f7993bedc7e98f68c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:40.329531 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.320119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45291 (local address 127.0.0.1:39170)
0620 04:30:40.321979 (+ 1860us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.321990 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.322070 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.322485 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.322488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.322506 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:40.322616 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.322623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.323786 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.323790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.324884 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.324895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.329058 (+ 4163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.329068 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.329157 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.329179 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.329252 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.329386 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1806,"thread_start_us":79,"threads_started":1}
I20260620 04:30:40.329826 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.320654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.0.98.1:45291)
0620 04:30:40.321046 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.321062 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.322081 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 04:30:40.322169 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.322175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.322363 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.322501 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:40.322727 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.323671 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.325026 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.325829 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.325929 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.326059 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.329303 (+ 3244us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.329509 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":80,"threads_started":1}
I20260620 04:30:40.332330 7404 tablet_service.cc:1467] Processing CreateTablet for tablet f6161a7e51094f7993bedc7e98f68c27 (DEFAULT_TABLE table=test-table [id=ec3ed84f9b31464e88a9e9f3cbbec566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:40.333264 7404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6161a7e51094f7993bedc7e98f68c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:40.337984 7780 tablet_bootstrap.cc:492] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: Bootstrap starting.
I20260620 04:30:40.339232 7779 tablet_bootstrap.cc:492] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: Bootstrap starting.
I20260620 04:30:40.339830 7781 tablet_bootstrap.cc:492] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: Bootstrap starting.
I20260620 04:30:40.340484 7780 tablet_bootstrap.cc:654] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:40.341444 7780 log.cc:824] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:40.341516 7779 tablet_bootstrap.cc:654] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:40.342298 7779 log.cc:824] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:40.342551 7781 tablet_bootstrap.cc:654] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:40.343255 7781 log.cc:824] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:40.343456 7780 tablet_bootstrap.cc:492] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: No bootstrap required, opened a new log
I20260620 04:30:40.343647 7780 ts_tablet_manager.cc:1397] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:30:40.344240 7779 tablet_bootstrap.cc:492] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: No bootstrap required, opened a new log
I20260620 04:30:40.344421 7779 ts_tablet_manager.cc:1397] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:30:40.345142 7781 tablet_bootstrap.cc:492] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: No bootstrap required, opened a new log
I20260620 04:30:40.345342 7781 ts_tablet_manager.cc:1397] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 04:30:40.347226 7780 raft_consensus.cc:348] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:40.347497 7780 raft_consensus.cc:374] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:40.347551 7780 raft_consensus.cc:729] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2919829f57ac4813af09f80a642e8f97, State: Initialized, Role: FOLLOWER
I20260620 04:30:40.348004 7780 consensus_queue.cc:260] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:40.348980 7780 ts_tablet_manager.cc:1428] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 04:30:40.348917 7779 raft_consensus.cc:348] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:40.349188 7779 raft_consensus.cc:374] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:40.349238 7779 raft_consensus.cc:729] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50af5eaac37b4e4690ea841a0a67096a, State: Initialized, Role: FOLLOWER
I20260620 04:30:40.349614 7746 heartbeater.cc:503] Master 127.0.98.62:37507 was elected leader, sending a full tablet report...
I20260620 04:30:40.349630 7781 raft_consensus.cc:348] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:40.349865 7781 raft_consensus.cc:374] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:40.349910 7781 raft_consensus.cc:729] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac4c91bd5224017a643817100960103, State: Initialized, Role: FOLLOWER
I20260620 04:30:40.350380 7781 consensus_queue.cc:260] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:40.351533 7781 ts_tablet_manager.cc:1428] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:30:40.352416 7779 consensus_queue.cc:260] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:40.352743 7471 heartbeater.cc:503] Master 127.0.98.62:37507 was elected leader, sending a full tablet report...
I20260620 04:30:40.353485 7779 ts_tablet_manager.cc:1428] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260620 04:30:40.353859 7608 heartbeater.cc:503] Master 127.0.98.62:37507 was elected leader, sending a full tablet report...
I20260620 04:30:40.471621 7785 raft_consensus.cc:482] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:40.471858 7785 raft_consensus.cc:504] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:40.473148 7785 leader_election.cc:288] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bac4c91bd5224017a643817100960103 (127.0.98.1:45291), 50af5eaac37b4e4690ea841a0a67096a (127.0.98.2:35869)
I20260620 04:30:40.477324 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.473378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45291 (local address 127.0.98.3:60821)
0620 04:30:40.473501 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.473517 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.473615 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.473907 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.473910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.473925 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:40.474049 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.474056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.474791 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.474795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.475865 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.475877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.476847 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.476851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.476947 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.476968 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.477062 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.477155 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":5}
I20260620 04:30:40.477299 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.473480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60821 (local address 127.0.98.1:45291)
0620 04:30:40.473676 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.473680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.473695 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:40.473748 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.473751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.473800 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.473876 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:40.474161 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.474682 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.476032 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.476653 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.476818 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.476882 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.477078 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.477163 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260620 04:30:40.477392 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.473685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56445 (local address 127.0.98.2:35869)
0620 04:30:40.473791 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.473795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.473806 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:30:40.473860 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.473863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.473912 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.473999 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:40.474228 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.474709 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.476155 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.476805 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.476912 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.476973 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.477172 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.477255 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:30:40.477345 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.473431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35869 (local address 127.0.98.3:56445)
0620 04:30:40.473622 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:40.473637 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:40.473720 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:40.474016 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:40.474018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:40.474032 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:40.474127 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.474133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.474810 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.474812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.476021 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:40.476031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.476922 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.476925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:40.477052 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.477074 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:30:40.477142 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:30:40.477208 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":20}
I20260620 04:30:40.477972 7424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6161a7e51094f7993bedc7e98f68c27" candidate_uuid: "2919829f57ac4813af09f80a642e8f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac4c91bd5224017a643817100960103" is_pre_election: true
I20260620 04:30:40.478222 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6161a7e51094f7993bedc7e98f68c27" candidate_uuid: "2919829f57ac4813af09f80a642e8f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50af5eaac37b4e4690ea841a0a67096a" is_pre_election: true
I20260620 04:30:40.478286 7424 raft_consensus.cc:2454] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2919829f57ac4813af09f80a642e8f97 in term 0.
I20260620 04:30:40.478513 7561 raft_consensus.cc:2454] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2919829f57ac4813af09f80a642e8f97 in term 0.
I20260620 04:30:40.478816 7630 leader_election.cc:302] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2919829f57ac4813af09f80a642e8f97, bac4c91bd5224017a643817100960103; no voters:
I20260620 04:30:40.479086 7785 raft_consensus.cc:2781] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:40.479172 7785 raft_consensus.cc:482] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:40.479205 7785 raft_consensus.cc:3037] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:40.480283 7785 raft_consensus.cc:504] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:40.480736 7785 leader_election.cc:288] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [CANDIDATE]: Term 1 election: Requested vote from peers bac4c91bd5224017a643817100960103 (127.0.98.1:45291), 50af5eaac37b4e4690ea841a0a67096a (127.0.98.2:35869)
I20260620 04:30:40.481103 7424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6161a7e51094f7993bedc7e98f68c27" candidate_uuid: "2919829f57ac4813af09f80a642e8f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac4c91bd5224017a643817100960103"
I20260620 04:30:40.481140 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6161a7e51094f7993bedc7e98f68c27" candidate_uuid: "2919829f57ac4813af09f80a642e8f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50af5eaac37b4e4690ea841a0a67096a"
I20260620 04:30:40.481216 7424 raft_consensus.cc:3037] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:40.481266 7561 raft_consensus.cc:3037] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:40.482221 7424 raft_consensus.cc:2454] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2919829f57ac4813af09f80a642e8f97 in term 1.
I20260620 04:30:40.482282 7561 raft_consensus.cc:2454] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2919829f57ac4813af09f80a642e8f97 in term 1.
I20260620 04:30:40.482486 7630 leader_election.cc:302] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2919829f57ac4813af09f80a642e8f97, bac4c91bd5224017a643817100960103; no voters:
I20260620 04:30:40.482647 7785 raft_consensus.cc:2781] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:40.482878 7785 raft_consensus.cc:686] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 1 LEADER]: Becoming Leader. State: Replica: 2919829f57ac4813af09f80a642e8f97, State: Running, Role: LEADER
I20260620 04:30:40.483222 7785 consensus_queue.cc:237] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:40.487115 6507 catalog_manager.cc:5495] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2919829f57ac4813af09f80a642e8f97 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2919829f57ac4813af09f80a642e8f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } health_report { overall_health: HEALTHY } } }
W20260620 04:30:40.520711 7748 tablet.cc:2369] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:30:40.528962 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.525174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.0.98.3:42819)
0620 04:30:40.525369 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.525374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.525389 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:40.525474 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.525478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.525549 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.525665 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:40.525927 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.526591 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.527507 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.527726 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.527819 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.528075 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:40.528604 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:40.528708 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.528755 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.528813 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 04:30:40.534987 7561 raft_consensus.cc:1261] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a [term 1 FOLLOWER]: Refusing update from remote peer 2919829f57ac4813af09f80a642e8f97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:40.534987 7424 raft_consensus.cc:1261] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103 [term 1 FOLLOWER]: Refusing update from remote peer 2919829f57ac4813af09f80a642e8f97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:40.535739 7788 consensus_queue.cc:1035] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:40.535918 7785 consensus_queue.cc:1035] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 04:30:40.539405 7610 tablet.cc:2369] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:30:40.544387 7791 mvcc.cc:205] Tried to move back new op lower bound from 7298784626824916992 to 7298784626622611456. Current Snapshot: MvccSnapshot[applied={T|T < 7298784626824916992}]
I20260620 04:30:40.549144 7792 mvcc.cc:205] Tried to move back new op lower bound from 7298784626824916992 to 7298784626622611456. Current Snapshot: MvccSnapshot[applied={T|T < 7298784626824916992}]
I20260620 04:30:40.555557 6507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:40.555761 6507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:40.560849 6507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c6740c79de24b00b6ec13d810e271bc: Sending DeleteTablet for 3 replicas of tablet f6161a7e51094f7993bedc7e98f68c27
I20260620 04:30:40.561657 7541 tablet_service.cc:1514] Processing DeleteTablet for tablet f6161a7e51094f7993bedc7e98f68c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:40 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54658
I20260620 04:30:40.561741 7404 tablet_service.cc:1514] Processing DeleteTablet for tablet f6161a7e51094f7993bedc7e98f68c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:40 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39170
I20260620 04:30:40.562199 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet f6161a7e51094f7993bedc7e98f68c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:40 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43264
I20260620 04:30:40.562434 7802 tablet_replica.cc:330] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: stopping tablet replica
I20260620 04:30:40.562678 7801 tablet_replica.cc:330] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: stopping tablet replica
I20260620 04:30:40.562798 7802 raft_consensus.cc:2229] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:40.562999 7801 raft_consensus.cc:2229] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:40.563140 7802 raft_consensus.cc:2258] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:40.563311 7801 raft_consensus.cc:2258] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:40.564939 7801 ts_tablet_manager.cc:1905] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:40.564939 7802 ts_tablet_manager.cc:1905] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:40.565377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 6475
I20260620 04:30:40.568300 7801 ts_tablet_manager.cc:1918] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:40.568326 7802 ts_tablet_manager.cc:1918] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:40.568397 7801 log.cc:1196] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/f6161a7e51094f7993bedc7e98f68c27
I20260620 04:30:40.568413 7802 log.cc:1196] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/f6161a7e51094f7993bedc7e98f68c27
I20260620 04:30:40.568689 7801 ts_tablet_manager.cc:1939] T f6161a7e51094f7993bedc7e98f68c27 P 50af5eaac37b4e4690ea841a0a67096a: Deleting consensus metadata
I20260620 04:30:40.568697 7802 ts_tablet_manager.cc:1939] T f6161a7e51094f7993bedc7e98f68c27 P bac4c91bd5224017a643817100960103: Deleting consensus metadata
W20260620 04:30:40.572095 7494 connection.cc:535] server connection from 127.0.0.1:54658 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 04:30:40.572539 7360 connection.cc:535] server connection from 127.0.0.1:39170 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:30:40.573226 7803 tablet_replica.cc:330] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: stopping tablet replica
I20260620 04:30:40.573572 7803 raft_consensus.cc:2229] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:40.574059 7803 raft_consensus.cc:2258] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:40.575745 7803 ts_tablet_manager.cc:1905] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:40.578886 7803 ts_tablet_manager.cc:1918] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:40.578989 7803 log.cc:1196] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/f6161a7e51094f7993bedc7e98f68c27
I20260620 04:30:40.579305 7803 ts_tablet_manager.cc:1939] T f6161a7e51094f7993bedc7e98f68c27 P 2919829f57ac4813af09f80a642e8f97: Deleting consensus metadata
W20260620 04:30:40.580394 7633 connection.cc:441] server connection from 127.0.0.1:43264 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43264 (request call id 1) could send its response
I20260620 04:30:40.583828 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.563767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.98.61:45881)
0620 04:30:40.563987 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.563992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.572939 (+ 8947us) server_negotiation.cc:407] Connection header received
0620 04:30:40.573197 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.573201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.573260 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.573330 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.574554 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.575290 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.576551 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.576920 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.577048 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.580364 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.580368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.580371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.580401 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.582082 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.582559 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.582563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.582564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.582649 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.582926 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.582929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.582931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.583123 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.583216 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.583555 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.583671 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260620 04:30:40.589830 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.575144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.0.98.61:45881)
0620 04:30:40.575247 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.575252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.580373 (+ 5121us) server_negotiation.cc:407] Connection header received
0620 04:30:40.580521 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.580524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.580566 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.580614 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.581437 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.582193 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.583472 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.583760 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.583841 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.585943 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.585947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.585950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.585972 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.587482 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.587964 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.587968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.587970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.588056 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.588329 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.588332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.588334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.588754 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.588933 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.589519 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.589663 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:30:40.590617 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.576858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.0.98.60:39179)
0620 04:30:40.581189 (+ 4331us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.581197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.581212 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:40.581259 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.581262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.581308 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.581394 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.582166 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.583065 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.584413 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.584732 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.584841 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.586325 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.586329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.586332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.586358 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.588178 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.588932 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.588936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.588938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.589129 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.589430 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.589433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.589434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.589646 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.589765 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.590326 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.590448 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4248,"thread_start_us":85,"threads_started":1}
I20260620 04:30:40.595517 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:40.563868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.0.98.60:39179)
0620 04:30:40.564069 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:40.564073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:40.585964 (+ 21891us) server_negotiation.cc:407] Connection header received
0620 04:30:40.586114 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:40.586117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:40.586165 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:40.586214 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:40.586903 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.587650 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.589896 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:40.590200 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:40.590288 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:40.592442 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:40.592446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:40.592449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:40.592475 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:40.594128 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.594493 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.594496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.594498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.594577 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:40.594779 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:40.594782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:40.594784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:40.594981 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:40.595098 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:40.595251 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 04:30:40.595361 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"spinlock_wait_cycles":896}
I20260620 04:30:41.547519 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:41.546998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.1:48979)
0620 04:30:41.547283 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:41.547390 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
W20260620 04:30:41.547894 7471 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37507 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
I20260620 04:30:41.549857 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:41.549378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.2:57879)
0620 04:30:41.549682 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:41.549771 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":75,"threads_started":1}
W20260620 04:30:41.550256 7608 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37507 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
I20260620 04:30:41.550282 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:41.549823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.3:54601)
0620 04:30:41.550104 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:41.550184 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":71,"threads_started":1}
W20260620 04:30:41.550554 7746 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37507 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
I20260620 04:30:42.061372 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c6740c79de24b00b6ec13d810e271bc)
I20260620 04:30:42.061548 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c6740c79de24b00b6ec13d810e271bc)
I20260620 04:30:42.061554 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } }
I20260620 04:30:42.061753 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } }
I20260620 04:30:42.062114 7815 leader_election.cc:288] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507), 7eec4f1aa71e4e1585702f0c51bb0a9c (127.0.98.61:45881)
I20260620 04:30:42.062677 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.062284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42538)
0620 04:30:42.062483 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.062574 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":55,"threads_started":1}
W20260620 04:30:42.062866 7275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
I20260620 04:30:42.063355 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.062817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42554)
0620 04:30:42.063188 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.063264 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":47,"threads_started":1}
W20260620 04:30:42.063547 7091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
I20260620 04:30:42.063845 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.063443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42560)
0620 04:30:42.063723 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.063775 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260620 04:30:42.063864 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507), f84c158313d640aab3c9cf26b3e54457 (127.0.98.60:39179)
W20260620 04:30:42.063951 7275 leader_election.cc:334] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
I20260620 04:30:42.064666 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.064416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42572)
0620 04:30:42.064546 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.064593 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 04:30:42.064756 7091 leader_election.cc:334] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
I20260620 04:30:42.066901 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.062616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45881 (local address 127.0.0.1:51360)
0620 04:30:42.062804 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.062816 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:42.062917 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:42.063386 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:42.063390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:42.063413 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:42.063527 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:42.063534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:42.065025 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:42.065029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:42.065881 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:42.065890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:42.066550 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:42.066554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:42.066633 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:42.066657 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:30:42.066724 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:30:42.066784 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260620 04:30:42.066923 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.062750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.0.98.61:45881)
0620 04:30:42.063138 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:42.063143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:42.063157 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:42.063204 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:42.063207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:42.063255 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:42.063754 (+ 499us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:42.063857 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:42.064884 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:42.066001 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:42.066448 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:42.066521 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:42.066576 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:42.066746 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 04:30:42.066812 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":78,"threads_started":1}
I20260620 04:30:42.067008 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.063557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.0.98.60:39179)
0620 04:30:42.064047 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:42.064057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:42.064070 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:42.064108 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:42.064111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:42.064149 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:42.064255 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:42.064478 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:42.065149 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:42.066171 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:42.066570 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:42.066660 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:42.066708 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:42.066861 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 04:30:42.066918 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":91,"threads_started":1}
I20260620 04:30:42.067029 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.063206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39179 (local address 127.0.0.1:37758)
0620 04:30:42.063487 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.063499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:42.063583 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:42.064266 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:42.064269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:42.064284 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:42.064379 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:42.064386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:42.065237 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:42.065239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:42.066033 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:42.066042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:42.066670 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:42.066673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:42.066752 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:42.066772 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:30:42.066842 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:30:42.066898 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":44,"threads_started":1}
I20260620 04:30:42.067319 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f84c158313d640aab3c9cf26b3e54457" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" is_pre_election: true
I20260620 04:30:42.067366 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f84c158313d640aab3c9cf26b3e54457" is_pre_election: true
I20260620 04:30:42.067474 7117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84c158313d640aab3c9cf26b3e54457 in term 1.
I20260620 04:30:42.067559 7296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eec4f1aa71e4e1585702f0c51bb0a9c in term 1.
I20260620 04:30:42.067742 7275 leader_election.cc:302] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eec4f1aa71e4e1585702f0c51bb0a9c, f84c158313d640aab3c9cf26b3e54457; no voters: 0c6740c79de24b00b6ec13d810e271bc
I20260620 04:30:42.067909 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 04:30:42.067970 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0c6740c79de24b00b6ec13d810e271bc)
I20260620 04:30:42.068022 7815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:30:42.068080 7092 leader_election.cc:302] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eec4f1aa71e4e1585702f0c51bb0a9c, f84c158313d640aab3c9cf26b3e54457; no voters: 0c6740c79de24b00b6ec13d810e271bc
I20260620 04:30:42.068286 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 04:30:42.068341 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0c6740c79de24b00b6ec13d810e271bc)
I20260620 04:30:42.068383 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:30:42.069146 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } }
I20260620 04:30:42.069370 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } }
I20260620 04:30:42.069451 7815 leader_election.cc:288] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [CANDIDATE]: Term 2 election: Requested vote from peers 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507), 7eec4f1aa71e4e1585702f0c51bb0a9c (127.0.98.61:45881)
I20260620 04:30:42.069806 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.069594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42574)
0620 04:30:42.069691 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.069742 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 04:30:42.069794 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 2 election: Requested vote from peers 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507), f84c158313d640aab3c9cf26b3e54457 (127.0.98.60:39179)
I20260620 04:30:42.069937 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f84c158313d640aab3c9cf26b3e54457" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c"
I20260620 04:30:42.070050 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.069861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42588)
0620 04:30:42.069942 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.069991 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 04:30:42.070083 7117 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f84c158313d640aab3c9cf26b3e54457 in current term 2: Already voted for candidate 7eec4f1aa71e4e1585702f0c51bb0a9c in this term.
I20260620 04:30:42.070456 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f84c158313d640aab3c9cf26b3e54457"
I20260620 04:30:42.070611 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.070425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42604)
0620 04:30:42.070503 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.070546 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 04:30:42.070621 7296 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7eec4f1aa71e4e1585702f0c51bb0a9c in current term 2: Already voted for candidate f84c158313d640aab3c9cf26b3e54457 in this term.
I20260620 04:30:42.070686 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.070512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42608)
0620 04:30:42.070592 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.070634 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 04:30:42.070693 7091 leader_election.cc:334] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
W20260620 04:30:42.070871 7275 leader_election.cc:334] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
I20260620 04:30:42.070924 7275 leader_election.cc:302] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f84c158313d640aab3c9cf26b3e54457; no voters: 0c6740c79de24b00b6ec13d810e271bc, 7eec4f1aa71e4e1585702f0c51bb0a9c
I20260620 04:30:42.070951 7092 leader_election.cc:302] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eec4f1aa71e4e1585702f0c51bb0a9c; no voters: 0c6740c79de24b00b6ec13d810e271bc, f84c158313d640aab3c9cf26b3e54457
I20260620 04:30:42.071038 7815 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 04:30:42.071133 7816 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 04:30:42.549118 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.548597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.1:38041)
0620 04:30:42.548892 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.548970 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260620 04:30:42.551156 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.550776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.2:33835)
0620 04:30:42.551019 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.551074 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20260620 04:30:42.551427 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:42.551001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.3:40945)
0620 04:30:42.551202 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:42.551344 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
I20260620 04:30:43.550750 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.550261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.1:42419)
0620 04:30:43.550564 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.550638 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
I20260620 04:30:43.552281 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.551727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.2:34369)
0620 04:30:43.552130 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.552188 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":69,"threads_started":1}
I20260620 04:30:43.552490 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.552125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.3:48167)
0620 04:30:43.552335 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.552395 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260620 04:30:43.812361 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:43.812500 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } }
I20260620 04:30:43.812907 7831 leader_election.cc:288] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507), f84c158313d640aab3c9cf26b3e54457 (127.0.98.60:39179)
I20260620 04:30:43.813366 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f84c158313d640aab3c9cf26b3e54457" is_pre_election: true
I20260620 04:30:43.813468 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.813089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42860)
0620 04:30:43.813320 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.813378 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":62,"threads_started":1}
I20260620 04:30:43.813499 7296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eec4f1aa71e4e1585702f0c51bb0a9c in term 2.
I20260620 04:30:43.813784 7092 leader_election.cc:302] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eec4f1aa71e4e1585702f0c51bb0a9c, f84c158313d640aab3c9cf26b3e54457; no voters:
I20260620 04:30:43.813942 7831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 04:30:43.813994 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:43.814064 7831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 2 FOLLOWER]: Advancing to term 3
I20260620 04:30:43.814213 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.813999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42864)
0620 04:30:43.814095 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.814146 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 04:30:43.814316 7091 leader_election.cc:334] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
I20260620 04:30:43.815136 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } }
I20260620 04:30:43.815428 7831 leader_election.cc:288] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 3 election: Requested vote from peers 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507), f84c158313d640aab3c9cf26b3e54457 (127.0.98.60:39179)
I20260620 04:30:43.815742 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.815530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42868)
0620 04:30:43.815628 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.815683 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 04:30:43.815789 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f84c158313d640aab3c9cf26b3e54457"
I20260620 04:30:43.815910 7296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 2 FOLLOWER]: Advancing to term 3
I20260620 04:30:43.816313 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.816123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42884)
0620 04:30:43.816215 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.816261 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 04:30:43.816416 7091 leader_election.cc:334] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
I20260620 04:30:43.816991 7296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eec4f1aa71e4e1585702f0c51bb0a9c in term 3.
I20260620 04:30:43.817257 7092 leader_election.cc:302] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eec4f1aa71e4e1585702f0c51bb0a9c, f84c158313d640aab3c9cf26b3e54457; no voters: 0c6740c79de24b00b6ec13d810e271bc
I20260620 04:30:43.817389 7831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 04:30:43.817541 7831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [term 3 LEADER]: Becoming Leader. State: Replica: 7eec4f1aa71e4e1585702f0c51bb0a9c, State: Running, Role: LEADER
I20260620 04:30:43.817775 7831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } }
I20260620 04:30:43.818732 7834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eec4f1aa71e4e1585702f0c51bb0a9c. Latest consensus state: current_term: 3 leader_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:43.818818 7834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:43.819051 7836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:43.821197 7836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:43.821550 7836 catalog_manager.cc:1259] Loaded cluster ID: db6e4e50e6564cd596f8c627a50e4c1b
I20260620 04:30:43.821590 7836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:43.822080 7836 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:43.822427 7836 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Loaded TSK: 0
I20260620 04:30:43.823043 7836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:30:43.899003 7099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:43.902249 7099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:43.905107 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.904844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42896)
0620 04:30:43.904951 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.905026 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 04:30:43.905259 7296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [term 3 FOLLOWER]: Refusing update from remote peer 7eec4f1aa71e4e1585702f0c51bb0a9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 04:30:43.905786 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.905599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:42900)
0620 04:30:43.905681 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.905725 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 04:30:43.905879 7091 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c -> Peer 0c6740c79de24b00b6ec13d810e271bc (127.0.98.62:37507): Couldn't send request to peer 0c6740c79de24b00b6ec13d810e271bc. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 04:30:43.905912 7831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 04:30:43.907362 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eec4f1aa71e4e1585702f0c51bb0a9c. Latest consensus state: current_term: 3 leader_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:43.907492 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:30:43.908474 7834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:43.908615 7834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:43.909397 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0c6740c79de24b00b6ec13d810e271bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "7eec4f1aa71e4e1585702f0c51bb0a9c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45881 } attrs { promote: false } } peers { permanent_uuid: "f84c158313d640aab3c9cf26b3e54457" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39179 } attrs { promote: false } } }
I20260620 04:30:43.909564 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84c158313d640aab3c9cf26b3e54457 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:30:43.919916 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.915862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42819 (local address 127.0.0.1:43290)
0620 04:30:43.916227 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.916239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:43.916299 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:43.916674 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:43.916677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:43.916692 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:43.916776 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:43.916781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.917545 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.917548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:43.918631 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:43.918640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.919517 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.919520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:43.919606 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:43.919632 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 04:30:43.919702 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:30:43.919772 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":79,"threads_started":1}
I20260620 04:30:43.919950 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.916202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.0.98.3:42819)
0620 04:30:43.916473 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:43.916477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:43.916489 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:30:43.916537 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:43.916539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:43.916578 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:43.916649 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:43.916873 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.917446 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.918756 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.919413 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.919524 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:43.919599 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:43.919718 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 04:30:43.919798 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":67,"threads_started":1}
I20260620 04:30:43.920580 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.915224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35869 (local address 127.0.0.1:54662)
0620 04:30:43.916073 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.916086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:43.916176 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:43.916471 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:43.916474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:43.916495 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:43.916609 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:43.916615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.917703 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.917706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:43.919302 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:43.919312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.920226 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.920229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:43.920312 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:43.920332 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:30:43.920389 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 04:30:43.920454 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":86,"threads_started":1}
I20260620 04:30:43.920648 7679 tablet_service.cc:1467] Processing CreateTablet for tablet ecefcf6d499640d6a7a4aaf0d882a6aa (DEFAULT_TABLE table=test-table [id=9bc285820fcf4d479199b8d691b0f4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:43.920946 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.915533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.0.98.2:35869)
0620 04:30:43.915780 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:43.915786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:43.916256 (+ 470us) server_negotiation.cc:407] Connection header received
0620 04:30:43.916312 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:43.916315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:43.916363 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:43.916440 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:43.916747 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.917605 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.919413 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.920006 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.920107 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:43.920173 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:43.920738 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 04:30:43.920819 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260620 04:30:43.920965 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecefcf6d499640d6a7a4aaf0d882a6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:43.921465 7541 tablet_service.cc:1467] Processing CreateTablet for tablet ecefcf6d499640d6a7a4aaf0d882a6aa (DEFAULT_TABLE table=test-table [id=9bc285820fcf4d479199b8d691b0f4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:43.921762 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecefcf6d499640d6a7a4aaf0d882a6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:43.923861 7848 tablet_bootstrap.cc:492] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a: Bootstrap starting.
I20260620 04:30:43.924373 7849 tablet_bootstrap.cc:492] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97: Bootstrap starting.
I20260620 04:30:43.924635 7848 tablet_bootstrap.cc:654] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:43.925391 7849 tablet_bootstrap.cc:654] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:43.925879 7848 tablet_bootstrap.cc:492] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a: No bootstrap required, opened a new log
I20260620 04:30:43.925967 7848 ts_tablet_manager.cc:1397] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:43.926437 7848 raft_consensus.cc:348] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:43.926537 7848 raft_consensus.cc:374] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:43.926565 7848 raft_consensus.cc:729] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50af5eaac37b4e4690ea841a0a67096a, State: Initialized, Role: FOLLOWER
I20260620 04:30:43.926729 7849 tablet_bootstrap.cc:492] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97: No bootstrap required, opened a new log
I20260620 04:30:43.926715 7848 consensus_queue.cc:260] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:43.926818 7849 ts_tablet_manager.cc:1397] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:43.927063 7848 ts_tablet_manager.cc:1428] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:43.927237 7849 raft_consensus.cc:348] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:43.927336 7849 raft_consensus.cc:374] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:43.927372 7849 raft_consensus.cc:729] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2919829f57ac4813af09f80a642e8f97, State: Initialized, Role: FOLLOWER
I20260620 04:30:43.927488 7849 consensus_queue.cc:260] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:43.927738 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.927455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.2:35257)
0620 04:30:43.927608 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.927666 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 04:30:43.927819 7849 ts_tablet_manager.cc:1428] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:43.928464 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.928237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.2:55223)
0620 04:30:43.928362 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.928408 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 04:30:43.929237 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.928765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.3:55481)
0620 04:30:43.929098 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.929167 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258}
I20260620 04:30:43.930264 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.929625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.3:50231)
0620 04:30:43.930152 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.930201 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450}
I20260620 04:30:43.930375 7747 heartbeater.cc:503] Master 127.0.98.61:45881 was elected leader, sending a full tablet report...
I20260620 04:30:43.930855 7609 heartbeater.cc:503] Master 127.0.98.61:45881 was elected leader, sending a full tablet report...
I20260620 04:30:43.933779 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.915014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45291 (local address 127.0.0.1:39180)
0620 04:30:43.915449 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.915464 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:43.915555 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:43.925445 (+ 9890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:43.925448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:43.925471 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:43.925580 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:43.925588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.931367 (+ 5779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.931371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:43.932482 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:43.932495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.933321 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.933324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:43.933421 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:43.933445 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:30:43.933514 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 04:30:43.933642 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370}
I20260620 04:30:43.933766 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.915100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.0.98.1:45291)
0620 04:30:43.925169 (+ 10069us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:43.925174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:43.925191 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:30:43.925246 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:43.925249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:43.925301 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:43.925705 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:43.925811 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.926522 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.932641 (+ 6119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:43.933186 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:43.933293 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:43.933366 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:43.933541 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 04:30:43.933623 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9981,"thread_start_us":67,"threads_started":1}
I20260620 04:30:43.934373 7404 tablet_service.cc:1467] Processing CreateTablet for tablet ecefcf6d499640d6a7a4aaf0d882a6aa (DEFAULT_TABLE table=test-table [id=9bc285820fcf4d479199b8d691b0f4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:43.934677 7404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecefcf6d499640d6a7a4aaf0d882a6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:43.936841 7854 tablet_bootstrap.cc:492] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103: Bootstrap starting.
I20260620 04:30:43.937943 7854 tablet_bootstrap.cc:654] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:43.939231 7854 tablet_bootstrap.cc:492] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103: No bootstrap required, opened a new log
I20260620 04:30:43.939323 7854 ts_tablet_manager.cc:1397] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:43.939759 7854 raft_consensus.cc:348] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:43.939869 7854 raft_consensus.cc:374] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:43.939890 7854 raft_consensus.cc:729] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac4c91bd5224017a643817100960103, State: Initialized, Role: FOLLOWER
I20260620 04:30:43.940022 7854 consensus_queue.cc:260] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:43.940362 7854 ts_tablet_manager.cc:1428] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:43.941054 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.940755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.1:42953)
0620 04:30:43.940874 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.940934 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 04:30:43.941772 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:43.941525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.1:59477)
0620 04:30:43.941671 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:43.941720 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 04:30:43.942389 7472 heartbeater.cc:503] Master 127.0.98.61:45881 was elected leader, sending a full tablet report...
I20260620 04:30:44.061683 7856 raft_consensus.cc:482] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:44.061839 7856 raft_consensus.cc:504] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:44.062950 7856 leader_election.cc:288] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50af5eaac37b4e4690ea841a0a67096a (127.0.98.2:35869), 2919829f57ac4813af09f80a642e8f97 (127.0.98.3:42819)
I20260620 04:30:44.068423 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.063281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58885 (local address 127.0.98.2:35869)
0620 04:30:44.063422 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:44.063426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:44.063709 (+ 283us) server_negotiation.cc:407] Connection header received
0620 04:30:44.063791 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:44.063793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:44.063836 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:44.063907 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:44.064364 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:44.065200 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:44.067021 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:44.067646 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:44.067756 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:44.067820 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:44.068221 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 04:30:44.068305 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:30:44.068461 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.063208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42819 (local address 127.0.98.1:58575)
0620 04:30:44.063519 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:44.063534 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:44.063662 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:44.063960 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:44.063963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:44.063997 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:44.064227 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:44.064235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:44.065338 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:44.065342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:44.066893 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:44.066906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:44.067812 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:44.067816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:44.068066 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:44.068088 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:30:44.068167 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 04:30:44.068264 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":264,"spinlock_wait_cycles":11392}
I20260620 04:30:44.068462 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.063367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58575 (local address 127.0.98.3:42819)
0620 04:30:44.063532 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:44.063537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:44.063605 (+ 68us) server_negotiation.cc:407] Connection header received
0620 04:30:44.063757 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:44.063761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:44.063815 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:44.063906 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:44.064364 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:44.065200 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:44.067019 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:44.067646 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:44.067756 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:44.067820 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:44.068194 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 04:30:44.068323 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:30:44.068845 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.063208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35869 (local address 127.0.98.1:58885)
0620 04:30:44.063617 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:44.063629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:44.063710 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:44.063960 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:44.063963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:44.063995 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:44.064233 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:44.064238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:44.065338 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:44.065342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:44.066893 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:44.066906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:44.067812 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:44.067816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:44.068038 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:44.068091 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 04:30:44.068634 (+ 543us) client_negotiation.cc:240] Negotiation successful
0620 04:30:44.068711 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":239,"spinlock_wait_cycles":3200,"thread_start_us":152,"threads_started":1}
I20260620 04:30:44.069068 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecefcf6d499640d6a7a4aaf0d882a6aa" candidate_uuid: "bac4c91bd5224017a643817100960103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2919829f57ac4813af09f80a642e8f97" is_pre_election: true
I20260620 04:30:44.069307 7699 raft_consensus.cc:2454] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac4c91bd5224017a643817100960103 in term 0.
I20260620 04:30:44.069293 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecefcf6d499640d6a7a4aaf0d882a6aa" candidate_uuid: "bac4c91bd5224017a643817100960103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50af5eaac37b4e4690ea841a0a67096a" is_pre_election: true
I20260620 04:30:44.069416 7561 raft_consensus.cc:2454] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac4c91bd5224017a643817100960103 in term 0.
I20260620 04:30:44.069788 7360 leader_election.cc:302] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2919829f57ac4813af09f80a642e8f97, bac4c91bd5224017a643817100960103; no voters:
I20260620 04:30:44.070073 7856 raft_consensus.cc:2781] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:44.070137 7856 raft_consensus.cc:482] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:44.070163 7856 raft_consensus.cc:3037] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:44.071173 7856 raft_consensus.cc:504] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:44.071520 7856 leader_election.cc:288] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [CANDIDATE]: Term 1 election: Requested vote from peers 50af5eaac37b4e4690ea841a0a67096a (127.0.98.2:35869), 2919829f57ac4813af09f80a642e8f97 (127.0.98.3:42819)
I20260620 04:30:44.071976 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecefcf6d499640d6a7a4aaf0d882a6aa" candidate_uuid: "bac4c91bd5224017a643817100960103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2919829f57ac4813af09f80a642e8f97"
I20260620 04:30:44.071954 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecefcf6d499640d6a7a4aaf0d882a6aa" candidate_uuid: "bac4c91bd5224017a643817100960103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50af5eaac37b4e4690ea841a0a67096a"
I20260620 04:30:44.072100 7699 raft_consensus.cc:3037] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:44.072135 7561 raft_consensus.cc:3037] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:44.073359 7699 raft_consensus.cc:2454] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac4c91bd5224017a643817100960103 in term 1.
I20260620 04:30:44.073329 7561 raft_consensus.cc:2454] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac4c91bd5224017a643817100960103 in term 1.
I20260620 04:30:44.073709 7359 leader_election.cc:302] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2919829f57ac4813af09f80a642e8f97, 50af5eaac37b4e4690ea841a0a67096a, bac4c91bd5224017a643817100960103; no voters:
I20260620 04:30:44.073891 7856 raft_consensus.cc:2781] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:44.074110 7856 raft_consensus.cc:686] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 1 LEADER]: Becoming Leader. State: Replica: bac4c91bd5224017a643817100960103, State: Running, Role: LEADER
I20260620 04:30:44.074364 7856 consensus_queue.cc:237] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } }
I20260620 04:30:44.074869 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.074617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.1:33945)
0620 04:30:44.074733 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:44.074794 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 04:30:44.077102 7107 catalog_manager.cc:5495] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 reported cstate change: term changed from 0 to 1, leader changed from <none> to bac4c91bd5224017a643817100960103 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bac4c91bd5224017a643817100960103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac4c91bd5224017a643817100960103" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 } health_report { overall_health: UNKNOWN } } }
W20260620 04:30:44.086716 7473 tablet.cc:2369] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:30:44.111409 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.108078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.0.98.1:45291)
0620 04:30:44.108221 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:44.108225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:44.108240 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:44.108318 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:44.108320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:44.108376 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:44.108469 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:44.108723 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:44.109297 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:44.110095 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:44.110333 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:44.110412 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:44.110619 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:44.111080 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:44.111193 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:44.111219 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 04:30:44.111270 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 04:30:44.114108 7561 raft_consensus.cc:1261] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a [term 1 FOLLOWER]: Refusing update from remote peer bac4c91bd5224017a643817100960103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:44.114375 7699 raft_consensus.cc:1261] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97 [term 1 FOLLOWER]: Refusing update from remote peer bac4c91bd5224017a643817100960103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:44.114648 7858 consensus_queue.cc:1035] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50af5eaac37b4e4690ea841a0a67096a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:44.114899 7856 consensus_queue.cc:1035] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2919829f57ac4813af09f80a642e8f97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:44.117980 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.117386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.3:46575)
0620 04:30:44.117831 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:44.117897 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371}
I20260620 04:30:44.118294 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.117715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.2:37767)
0620 04:30:44.118163 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:44.118219 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382}
I20260620 04:30:44.121197 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.120339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.3:41071)
0620 04:30:44.121065 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:44.121121 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638}
I20260620 04:30:44.121236 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.120940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.2:60413)
0620 04:30:44.121132 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:44.121179 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260620 04:30:44.122749 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.122357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.1:48461)
0620 04:30:44.122619 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:44.122675 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260620 04:30:44.123852 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.123651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.98.1:58019)
0620 04:30:44.123739 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:44.123790 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 04:30:44.128749 7099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:44.128940 7099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:44.132941 7099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eec4f1aa71e4e1585702f0c51bb0a9c: Sending DeleteTablet for 3 replicas of tablet ecefcf6d499640d6a7a4aaf0d882a6aa
I20260620 04:30:44.133869 7541 tablet_service.cc:1514] Processing DeleteTablet for tablet ecefcf6d499640d6a7a4aaf0d882a6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:44 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54662
I20260620 04:30:44.133882 7404 tablet_service.cc:1514] Processing DeleteTablet for tablet ecefcf6d499640d6a7a4aaf0d882a6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:44 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39180
I20260620 04:30:44.134024 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet ecefcf6d499640d6a7a4aaf0d882a6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:44 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43290
I20260620 04:30:44.134239 7872 tablet_replica.cc:330] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103: stopping tablet replica
I20260620 04:30:44.134259 7873 tablet_replica.cc:330] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a: stopping tablet replica
I20260620 04:30:44.134366 7873 raft_consensus.cc:2229] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:44.134366 7872 raft_consensus.cc:2229] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:44.134686 7872 raft_consensus.cc:2258] T ecefcf6d499640d6a7a4aaf0d882a6aa P bac4c91bd5224017a643817100960103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:44.134858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 7251
I20260620 04:30:44.135515 7874 tablet_replica.cc:330] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97: stopping tablet replica
I20260620 04:30:44.135643 7874 raft_consensus.cc:2229] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:44.135771 7874 raft_consensus.cc:2258] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:44.136618 7874 ts_tablet_manager.cc:1905] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:44.137341 7873 raft_consensus.cc:2258] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:44.138041 7873 ts_tablet_manager.cc:1905] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:44.139518 7874 ts_tablet_manager.cc:1918] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:44.139600 7874 log.cc:1196] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/ecefcf6d499640d6a7a4aaf0d882a6aa
I20260620 04:30:44.139876 7874 ts_tablet_manager.cc:1939] T ecefcf6d499640d6a7a4aaf0d882a6aa P 2919829f57ac4813af09f80a642e8f97: Deleting consensus metadata
I20260620 04:30:44.140574 7873 ts_tablet_manager.cc:1918] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:44.140654 7873 log.cc:1196] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/ecefcf6d499640d6a7a4aaf0d882a6aa
I20260620 04:30:44.140672 7094 catalog_manager.cc:4841] TS 2919829f57ac4813af09f80a642e8f97 (127.0.98.3:42819): tablet ecefcf6d499640d6a7a4aaf0d882a6aa (table test-table [id=9bc285820fcf4d479199b8d691b0f4e4]) successfully deleted
I20260620 04:30:44.140920 7873 ts_tablet_manager.cc:1939] T ecefcf6d499640d6a7a4aaf0d882a6aa P 50af5eaac37b4e4690ea841a0a67096a: Deleting consensus metadata
I20260620 04:30:44.141876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 7478
I20260620 04:30:44.141812 7093 catalog_manager.cc:4841] TS 50af5eaac37b4e4690ea841a0a67096a (127.0.98.2:35869): tablet ecefcf6d499640d6a7a4aaf0d882a6aa (table test-table [id=9bc285820fcf4d479199b8d691b0f4e4]) successfully deleted
I20260620 04:30:44.141945 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:44.141688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45291 (local address 127.0.0.1:39200)
0620 04:30:44.141805 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:44.141863 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 04:30:44.142817 7091 catalog_manager.cc:4584] TS bac4c91bd5224017a643817100960103 (127.0.98.1:45291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ecefcf6d499640d6a7a4aaf0d882a6aa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45291: connect: Connection refused (error 111)
I20260620 04:30:44.147639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 7614
I20260620 04:30:44.153792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 6971
I20260620 04:30:44.159106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 7079
2026-06-20T04:30:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7127 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:44 dist-test-slave-rzqm krb5kdc[7877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:44 dist-test-slave-rzqm krb5kdc[7877](info): set up 2 sockets
Jun 20 04:30:44 dist-test-slave-rzqm krb5kdc[7877](info): commencing operation
krb5kdc: starting...
W20260620 04:30:45.113713 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:45 dist-test-slave-rzqm krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 04:30:45.229992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43185
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:45.340736 7890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:45.341056 7890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:45.341118 7890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:45.345213 7890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:45.345273 7890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 04:30:45.345289 7890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:45.345306 7890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:45.345324 7890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:45.345353 7890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:45.350839 7890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43185
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:45.351969 7890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:45.353071 7890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:45.361568 7895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:45.361650 7890 server_base.cc:1034] running on GCE node
W20260620 04:30:45.361670 7896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:45.361579 7898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:45.362601 7890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:45.363885 7890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:45.365072 7890 hybrid_clock.cc:648] HybridClock initialized: now 1781929845365052 us; error 31 us; skew 500 ppm
Jun 20 04:30:45 dist-test-slave-rzqm krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:45.367865 7890 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 04:30:45.368911 7890 webserver.cc:460] Webserver started at http://127.0.98.62:35531/ using document root <none> and password file <none>
I20260620 04:30:45.369520 7890 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:45.369586 7890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:45.369757 7890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:45.371912 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "451ff04375174e2bae5ca68ab318fcde"
format_stamp: "Formatted at 2026-06-20 04:30:45 on dist-test-slave-rzqm"
server_key: "620a13dde9e6a7fd29f8884d483cd571"
server_key_iv: "2520f1e57fe2bc78e74f389c75d5d11a"
server_key_version: "encryptionkey@0"
I20260620 04:30:45.372452 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "451ff04375174e2bae5ca68ab318fcde"
format_stamp: "Formatted at 2026-06-20 04:30:45 on dist-test-slave-rzqm"
server_key: "620a13dde9e6a7fd29f8884d483cd571"
server_key_iv: "2520f1e57fe2bc78e74f389c75d5d11a"
server_key_version: "encryptionkey@0"
I20260620 04:30:45.376236 7890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:30:45.378930 7905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:45.380342 7890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 04:30:45.380475 7890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "451ff04375174e2bae5ca68ab318fcde"
format_stamp: "Formatted at 2026-06-20 04:30:45 on dist-test-slave-rzqm"
server_key: "620a13dde9e6a7fd29f8884d483cd571"
server_key_iv: "2520f1e57fe2bc78e74f389c75d5d11a"
server_key_version: "encryptionkey@0"
I20260620 04:30:45.380605 7890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:45.400852 7890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:45.404589 7890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:45.404783 7890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:45.412523 7890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43185
I20260620 04:30:45.412593 7967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43185 every 8 connection(s)
I20260620 04:30:45.413525 7890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:45.416137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 7890
I20260620 04:30:45.416244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:45.416337 7968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:45.416556 392 external_mini_cluster.cc:1428] Setting key 482039f7c3cc8dd703d2a2676216ff5b
I20260620 04:30:45.423209 7968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde: Bootstrap starting.
I20260620 04:30:45.425988 7968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde: Neither blocks nor log segments found. Creating new log.
Jun 20 04:30:45 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:30:45.426746 7968 log.cc:824] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:45.428656 7968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde: No bootstrap required, opened a new log
I20260620 04:30:45.431696 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:45.418013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.0.98.62:43185)
0620 04:30:45.418429 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:45.418445 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:45.418483 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:30:45.419448 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:45.419469 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:45.419838 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:45.420143 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:45.421090 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.422089 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.423257 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.423865 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.424000 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:45.426773 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:45.426778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:45.426793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:45.426824 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:45.428846 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:45.429327 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:45.429332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:45.429336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:45.429452 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:45.429733 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:45.429736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:45.429738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:45.430139 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:45.430349 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:45.430611 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 04:30:45.430912 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260620 04:30:45.431809 7968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451ff04375174e2bae5ca68ab318fcde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } }
I20260620 04:30:45.432113 7968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:45.432168 7968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451ff04375174e2bae5ca68ab318fcde, State: Initialized, Role: FOLLOWER
I20260620 04:30:45.432595 7968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451ff04375174e2bae5ca68ab318fcde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } }
I20260620 04:30:45.432715 7968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:45.432757 7968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:45.432849 7968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:45.433797 7968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451ff04375174e2bae5ca68ab318fcde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } }
I20260620 04:30:45.434166 7968 leader_election.cc:302] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451ff04375174e2bae5ca68ab318fcde; no voters:
I20260620 04:30:45.434509 7968 leader_election.cc:288] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:45.434602 7973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:45.434815 7973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [term 1 LEADER]: Becoming Leader. State: Replica: 451ff04375174e2bae5ca68ab318fcde, State: Running, Role: LEADER
I20260620 04:30:45.435150 7973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451ff04375174e2bae5ca68ab318fcde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } }
I20260620 04:30:45.435693 7968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:45.436748 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451ff04375174e2bae5ca68ab318fcde. Latest consensus state: current_term: 1 leader_uuid: "451ff04375174e2bae5ca68ab318fcde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451ff04375174e2bae5ca68ab318fcde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } } }
I20260620 04:30:45.436880 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:45.437234 7974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "451ff04375174e2bae5ca68ab318fcde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451ff04375174e2bae5ca68ab318fcde" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43185 } } }
I20260620 04:30:45.437258 7982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:45.437464 7974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:45.440275 7982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:45.445370 7982 catalog_manager.cc:1347] Generated new cluster ID: c9fe5ac0a4f04663bee019e4fa29e82a
I20260620 04:30:45.445457 7982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:45.463912 7982 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:45.464896 7982 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:45.470784 7982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde: Generated new TSK 0
I20260620 04:30:45.471541 7982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 04:30:45.504624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43185
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:45.616525 7994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:45.616820 7994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:45.616873 7994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:45.621130 7994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:45.621201 7994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:45.621241 7994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:30:45.621277 7994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:45.626724 7994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:45.627897 7994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:45.629030 7994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:45.639467 7999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:45.639472 8002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:45.639472 8000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:45.639528 7994 server_base.cc:1034] running on GCE node
I20260620 04:30:45.640264 7994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:45.640936 7994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:45.642114 7994 hybrid_clock.cc:648] HybridClock initialized: now 1781929845642102 us; error 27 us; skew 500 ppm
Jun 20 04:30:45 dist-test-slave-rzqm krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:45.645088 7994 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:30:45.646016 7994 webserver.cc:460] Webserver started at http://127.0.98.1:41775/ using document root <none> and password file <none>
I20260620 04:30:45.646585 7994 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:45.646654 7994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:45.646863 7994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:45.648977 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ee9801a81e84f21afe0f60f8e3c08b0"
format_stamp: "Formatted at 2026-06-20 04:30:45 on dist-test-slave-rzqm"
server_key: "ddd28375417a3449d670d6e5c8e6013b"
server_key_iv: "6d23fbbad009438e25480c0efae0f640"
server_key_version: "encryptionkey@0"
I20260620 04:30:45.649580 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ee9801a81e84f21afe0f60f8e3c08b0"
format_stamp: "Formatted at 2026-06-20 04:30:45 on dist-test-slave-rzqm"
server_key: "ddd28375417a3449d670d6e5c8e6013b"
server_key_iv: "6d23fbbad009438e25480c0efae0f640"
server_key_version: "encryptionkey@0"
I20260620 04:30:45.653497 7994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:45.656252 8009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:45.657662 7994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:45.657771 7994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "0ee9801a81e84f21afe0f60f8e3c08b0"
format_stamp: "Formatted at 2026-06-20 04:30:45 on dist-test-slave-rzqm"
server_key: "ddd28375417a3449d670d6e5c8e6013b"
server_key_iv: "6d23fbbad009438e25480c0efae0f640"
server_key_version: "encryptionkey@0"
I20260620 04:30:45.657892 7994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:45.680015 7994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:45.683333 7994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:45.683521 7994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:45.685372 7994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:45.685441 7994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:45.685493 7994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:45.685521 7994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:45.695487 7994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32887
Jun 20 04:30:45 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:30:45.695539 8124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32887 every 8 connection(s)
I20260620 04:30:45.696610 7994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:45.700220 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:45.687582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51349 (local address 127.0.98.62:43185)
0620 04:30:45.687716 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:45.687720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:45.687748 (+ 28us) server_negotiation.cc:407] Connection header received
0620 04:30:45.688531 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:45.688536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:45.688597 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:45.688654 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:45.690499 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.691353 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.692779 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.693163 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.693284 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:45.696515 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:45.696519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:45.696522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:45.696556 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:45.698143 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:45.698674 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:45.698678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:45.698680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:45.698777 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:45.699109 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:45.699113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:45.699115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:45.699316 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:45.699406 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:45.699912 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 04:30:45.700050 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:30:45.701064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 7994
I20260620 04:30:45.701188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:45.701166 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:45.686738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43185 (local address 127.0.98.1:51349)
0620 04:30:45.687269 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:45.687322 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:45.688277 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:45.689139 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:45.689150 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:45.689575 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:45.690306 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:45.690328 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.691506 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.691510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:45.692630 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:45.692644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.693346 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.693349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:45.693506 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:45.694172 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:45.694192 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:45.696289 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:45.698274 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:45.698281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:45.698298 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:45.698568 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:45.698862 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:45.698866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:45.698869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:45.699016 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:45.699414 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:45.699422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:45.699675 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 04:30:45.699854 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 04:30:45.700195 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260620 04:30:45.701534 392 external_mini_cluster.cc:1428] Setting key f7f8a95f6b501e63fc5afccfe2cc2b11
I20260620 04:30:45.702710 8125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43185
I20260620 04:30:45.702869 8125 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:45.703507 8125 heartbeater.cc:511] Master 127.0.98.62:43185 requested a full tablet report, sending...
I20260620 04:30:45.705374 7921 ts_manager.cc:194] Registered new tserver with Master: 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1:32887)
I20260620 04:30:45.706646 7921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51349
I20260620 04:30:45.712787 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:45.705030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45849 (local address 127.0.98.62:43185)
0620 04:30:45.705145 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:45.705150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:45.705328 (+ 178us) server_negotiation.cc:407] Connection header received
0620 04:30:45.705436 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:45.705438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:45.705476 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:45.705551 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:45.706031 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.706595 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.707669 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.707980 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.708086 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:45.709390 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:45.709393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:45.709396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:45.709423 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:45.710948 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:45.711498 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:45.711501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:45.711502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:45.711589 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:45.711917 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:45.711920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:45.711921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:45.712160 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:45.712250 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:45.712543 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 04:30:45.712660 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":45}
I20260620 04:30:45.712847 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:45.704939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43185 (local address 127.0.98.1:45849)
0620 04:30:45.705243 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:45.705259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:45.705361 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:45.705569 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:45.705572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:45.705786 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:45.705909 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:45.705916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.706725 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.706729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:45.707553 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:45.707565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.708097 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.708100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:45.708198 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:45.708791 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:45.708804 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:45.709253 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:45.711070 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:45.711074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:45.711076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:45.711377 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:45.711686 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:45.711690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:45.711692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:45.711794 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:45.712258 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:45.712262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:45.712374 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 04:30:45.712485 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 04:30:45.712638 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 04:30:45.732389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43185
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:45.846295 8134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:45.846575 8134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:45.846638 8134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:45.850868 8134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:45.850941 8134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:45.850999 8134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:30:45.851055 8134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:45.856601 8134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:45.857774 8134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:45.858839 8134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:45.868821 8142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:45.868821 8140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:45.868848 8139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:45.869215 8134 server_base.cc:1034] running on GCE node
I20260620 04:30:45.869764 8134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:45.870450 8134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:45.871628 8134 hybrid_clock.cc:648] HybridClock initialized: now 1781929845871598 us; error 48 us; skew 500 ppm
Jun 20 04:30:45 dist-test-slave-rzqm krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:45.874711 8134 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:30:45.875710 8134 webserver.cc:460] Webserver started at http://127.0.98.2:41969/ using document root <none> and password file <none>
I20260620 04:30:45.876276 8134 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:45.876335 8134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:45.876507 8134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:45.878868 8134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "bd6da02a19fc4f3ea8b25e330a2bf082"
format_stamp: "Formatted at 2026-06-20 04:30:45 on dist-test-slave-rzqm"
server_key: "0d579b265f0e03f8bf12939a96f3701b"
server_key_iv: "2428dfd6a7a4b368dfdbbddc0787ac60"
server_key_version: "encryptionkey@0"
I20260620 04:30:45.879428 8134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bd6da02a19fc4f3ea8b25e330a2bf082"
format_stamp: "Formatted at 2026-06-20 04:30:45 on dist-test-slave-rzqm"
server_key: "0d579b265f0e03f8bf12939a96f3701b"
server_key_iv: "2428dfd6a7a4b368dfdbbddc0787ac60"
server_key_version: "encryptionkey@0"
I20260620 04:30:45.883369 8134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 04:30:45.886175 8149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:45.887555 8134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:45.887681 8134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "bd6da02a19fc4f3ea8b25e330a2bf082"
format_stamp: "Formatted at 2026-06-20 04:30:45 on dist-test-slave-rzqm"
server_key: "0d579b265f0e03f8bf12939a96f3701b"
server_key_iv: "2428dfd6a7a4b368dfdbbddc0787ac60"
server_key_version: "encryptionkey@0"
I20260620 04:30:45.887785 8134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:45.912019 8134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:45.915619 8134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:45.915793 8134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:45.917341 8134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:45.917402 8134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:45.917446 8134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:45.917477 8134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:30:45 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:30:45.927727 8134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45077
I20260620 04:30:45.927745 8264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45077 every 8 connection(s)
I20260620 04:30:45.928788 8134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:45.929198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 8134
I20260620 04:30:45.929316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:45.929636 392 external_mini_cluster.cc:1428] Setting key 277db10c752429d29538b9b0bcd95a31
I20260620 04:30:45.932029 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:45.918226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60273 (local address 127.0.98.62:43185)
0620 04:30:45.918374 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:45.918379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:45.919316 (+ 937us) server_negotiation.cc:407] Connection header received
0620 04:30:45.920421 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:45.920426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:45.920483 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:45.920533 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:45.922085 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.922651 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.924447 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.924823 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.924935 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:45.928021 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:45.928025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:45.928027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:45.928055 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:45.929688 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:45.930349 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:45.930352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:45.930353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:45.930449 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:45.930829 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:45.930831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:45.930833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:45.931017 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:45.931114 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:45.931783 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 04:30:45.931902 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:30:45.933244 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:45.918649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43185 (local address 127.0.98.2:60273)
0620 04:30:45.919138 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:45.919194 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:45.920161 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:45.920914 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:45.920921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:45.921343 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:45.921918 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:45.921934 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.922782 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.922786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:45.924284 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:45.924298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.925025 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.925031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:45.925255 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:45.926020 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:45.926043 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:45.927824 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:45.929837 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:45.929843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:45.929860 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:45.930221 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:45.930540 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:45.930543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:45.930545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:45.930739 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:45.931125 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:45.931132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:45.931473 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 04:30:45.931708 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 04:30:45.932098 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260620 04:30:45.934551 8265 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43185
I20260620 04:30:45.934754 8265 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:45.935366 8265 heartbeater.cc:511] Master 127.0.98.62:43185 requested a full tablet report, sending...
I20260620 04:30:45.936582 7922 ts_manager.cc:194] Registered new tserver with Master: bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077)
I20260620 04:30:45.937331 7922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60273
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260620 04:30:45.944948 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:45.937637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57115 (local address 127.0.98.62:43185)
0620 04:30:45.937861 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:45.937866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:45.937880 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:45.937916 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:45.937923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:45.937967 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:45.938043 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:45.938670 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.939453 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.940324 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.940546 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.940633 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:45.941830 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:45.941833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:45.941835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:45.941858 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:45.943309 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:45.943735 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:45.943739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:45.943741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:45.943821 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:45.944074 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:45.944077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:45.944079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:45.944311 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:45.944403 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:45.944671 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 04:30:45.944788 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260620 04:30:45.944962 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:45.936801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43185 (local address 127.0.98.2:57115)
0620 04:30:45.937142 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:45.937157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:45.937253 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:45.938160 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:45.938164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:45.938388 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:45.938528 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:45.938537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.939608 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.939610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:45.940207 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:45.940217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:45.940682 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:45.940685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:45.940755 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:45.941280 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:45.941293 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:45.941655 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:45.943425 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:45.943428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:45.943431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:45.943640 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:45.943906 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:45.943910 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:45.943912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:45.943994 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:45.944409 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:45.944413 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:45.944509 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 04:30:45.944597 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 04:30:45.944765 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":3,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 04:30:45.960222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43185
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:46.074091 8274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:46.074359 8274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:46.074404 8274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:46.078557 8274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:46.078625 8274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:46.078665 8274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 04:30:46.078697 8274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:46.084019 8274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:46.085141 8274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:46.086207 8274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:46.095580 8279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:46.095615 8282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:46.095657 8280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:46.095861 8274 server_base.cc:1034] running on GCE node
I20260620 04:30:46.096284 8274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:46.096925 8274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:46.098085 8274 hybrid_clock.cc:648] HybridClock initialized: now 1781929846098052 us; error 47 us; skew 500 ppm
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:46.100777 8274 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:30:46.101716 8274 webserver.cc:460] Webserver started at http://127.0.98.3:38611/ using document root <none> and password file <none>
I20260620 04:30:46.102255 8274 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:46.102321 8274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:46.102552 8274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:46.104724 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "810b9e5727f44bfc8e33ad717d1a6174"
format_stamp: "Formatted at 2026-06-20 04:30:46 on dist-test-slave-rzqm"
server_key: "f1576e75eb6bf1dc07bc65208a44296b"
server_key_iv: "6cdc83f02c1c2d256fb86b66ac13a84f"
server_key_version: "encryptionkey@0"
I20260620 04:30:46.105304 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "810b9e5727f44bfc8e33ad717d1a6174"
format_stamp: "Formatted at 2026-06-20 04:30:46 on dist-test-slave-rzqm"
server_key: "f1576e75eb6bf1dc07bc65208a44296b"
server_key_iv: "6cdc83f02c1c2d256fb86b66ac13a84f"
server_key_version: "encryptionkey@0"
I20260620 04:30:46.109112 8274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:30:46.111654 8289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:46.112941 8274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:46.113157 8274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "810b9e5727f44bfc8e33ad717d1a6174"
format_stamp: "Formatted at 2026-06-20 04:30:46 on dist-test-slave-rzqm"
server_key: "f1576e75eb6bf1dc07bc65208a44296b"
server_key_iv: "6cdc83f02c1c2d256fb86b66ac13a84f"
server_key_version: "encryptionkey@0"
I20260620 04:30:46.113265 8274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:46.130188 8274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:46.133633 8274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:46.133828 8274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:46.135401 8274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:46.135464 8274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:46.135527 8274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:46.135552 8274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:30:46.145951 8274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38677
I20260620 04:30:46.146030 8404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38677 every 8 connection(s)
I20260620 04:30:46.147045 8274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:46.149530 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.136200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44441 (local address 127.0.98.62:43185)
0620 04:30:46.136339 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.136344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.137364 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 04:30:46.138653 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.138657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.138717 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.138811 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:46.140477 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.141318 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.142655 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.142989 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.143102 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.145804 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:46.145808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:46.145811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:46.145840 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:46.147294 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:46.148023 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:46.148026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:46.148027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:46.148118 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:46.148430 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:46.148434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:46.148436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:46.148629 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:46.148714 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.149229 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.149369 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:30:46.150302 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.136666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43185 (local address 127.0.98.3:44441)
0620 04:30:46.137189 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.137253 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.138402 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.139018 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.139028 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.139528 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:46.140295 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.140318 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.141439 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.141443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.142500 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.142510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.143146 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.143150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.143328 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.143964 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:46.143984 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:46.145627 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:46.147404 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:46.147411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:46.147428 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:46.147759 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:46.148205 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:46.148208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:46.148210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:46.148345 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:46.148721 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:46.148728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:46.148968 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.149172 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.149498 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260620 04:30:46.151618 8405 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43185
I20260620 04:30:46.151772 8405 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:46.152315 8405 heartbeater.cc:511] Master 127.0.98.62:43185 requested a full tablet report, sending...
I20260620 04:30:46.153409 7921 ts_manager.cc:194] Registered new tserver with Master: 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677)
I20260620 04:30:46.154141 7921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44441
I20260620 04:30:46.156878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 8274
I20260620 04:30:46.157033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:46.157387 392 external_mini_cluster.cc:1428] Setting key db7d445fc141dbf62d964f0aa06e0341
I20260620 04:30:46.159986 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:30:46.162320 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.153687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54093 (local address 127.0.98.62:43185)
0620 04:30:46.153811 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.153815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.153991 (+ 176us) server_negotiation.cc:407] Connection header received
0620 04:30:46.154108 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.154110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.154148 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.154213 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:46.154698 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.155209 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.156175 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.156482 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.156590 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.157891 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:46.157895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:46.157898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:46.157927 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:46.159495 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:46.159939 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:46.159942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:46.159943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:46.160022 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:46.161321 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:46.161324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:46.161326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:46.161573 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:46.161680 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.162048 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.162156 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:30:46.162350 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.153653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43185 (local address 127.0.98.3:54093)
0620 04:30:46.153916 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.153931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.154023 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.154247 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.154249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.154459 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:46.154593 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.154601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.155343 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.155346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.156059 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.156068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.156635 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.156639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.156735 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.157280 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:46.157294 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:46.157756 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:46.159601 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:46.159605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:46.159607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:46.159858 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:46.161100 (+ 1242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:46.161103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:46.161105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:46.161209 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:46.161690 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:46.161693 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:46.161800 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.161983 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.162157 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929846, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:30:46.184899 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:46.178735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.0.98.62:43185)
0620 04:30:46.178845 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.178849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.178860 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:30:46.178899 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.178902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.178939 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.178998 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:46.179741 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.180348 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.181268 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.181476 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.181553 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.184744 (+ 3191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54238: BlockingRecv error: recv got EOF from 127.0.0.1:54238 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929846, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:30:46.194388 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.186467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.0.98.62:43185)
0620 04:30:46.186597 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.186601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.186730 (+ 129us) server_negotiation.cc:407] Connection header received
0620 04:30:46.186828 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.186830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.186868 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.186931 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:46.187451 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.187987 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.188772 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.189052 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.189186 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.191555 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:46.191559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:46.191561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:46.191588 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:46.192955 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:46.193399 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:46.193402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:46.193403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:46.193486 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:46.193699 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:46.193701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:46.193702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:46.193885 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:46.193980 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.194167 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.194262 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:30:46.197522 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:46.200554 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:46.215276 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.209456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38677 (local address 127.0.0.1:35070)
0620 04:30:46.210008 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.210060 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.210191 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.210777 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.210783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.210803 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:46.210951 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.210958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.212120 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.212123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.213546 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.213556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.214609 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.214612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.214713 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.214747 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.214942 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.215084 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":24,"thread_start_us":97,"threads_started":1}
I20260620 04:30:46.215276 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.209560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.0.98.3:38677)
0620 04:30:46.209915 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.209928 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.210142 (+ 214us) server_negotiation.cc:407] Connection header received
0620 04:30:46.210393 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.210399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.210635 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.210788 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:46.211081 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.212015 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.213687 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.214523 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.214624 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.214804 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.214963 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.215083 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":128,"threads_started":1}
I20260620 04:30:46.215778 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.210752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.0.98.1:32887)
0620 04:30:46.211145 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.211151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.211200 (+ 49us) server_negotiation.cc:407] Connection header received
0620 04:30:46.211277 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.211282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.211464 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.211612 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:46.211994 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.212746 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.214025 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.214582 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.214682 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.214819 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.215542 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.215647 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":61,"threads_started":1}
I20260620 04:30:46.215317 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.210522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32887 (local address 127.0.0.1:40608)
0620 04:30:46.211054 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.211067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.211139 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.211763 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.211766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.211784 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:46.211868 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.211875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.212882 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.212885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.213910 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.213919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.214880 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.214883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.214985 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.215008 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.215075 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.215147 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":84,"threads_started":1}
I20260620 04:30:46.216080 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.210207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45077 (local address 127.0.0.1:43668)
0620 04:30:46.210442 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.210507 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.210589 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.211595 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.211598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.211615 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:46.211717 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.211723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.213286 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.213289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.214441 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.214452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.215726 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.215730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.215818 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.215839 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.215900 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.215962 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":22,"thread_start_us":93,"threads_started":1}
I20260620 04:30:46.216416 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.210807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.0.98.2:45077)
0620 04:30:46.211234 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.211244 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.211262 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:30:46.211320 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.211324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.211472 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.211612 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:46.211813 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.213164 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.214578 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.215383 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.215718 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.216059 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.216133 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.216282 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":119,"threads_started":1}
I20260620 04:30:46.218556 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 80a1aa24e8534fddbe0a48bf0ddce598 (DEFAULT_TABLE table=test-table [id=66ea4b6d32624150b10b978dbde41e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.219565 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80a1aa24e8534fddbe0a48bf0ddce598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.219460 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 80a1aa24e8534fddbe0a48bf0ddce598 (DEFAULT_TABLE table=test-table [id=66ea4b6d32624150b10b978dbde41e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.219456 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 80a1aa24e8534fddbe0a48bf0ddce598 (DEFAULT_TABLE table=test-table [id=66ea4b6d32624150b10b978dbde41e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.220542 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80a1aa24e8534fddbe0a48bf0ddce598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.220542 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80a1aa24e8534fddbe0a48bf0ddce598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.226173 8433 tablet_bootstrap.cc:492] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: Bootstrap starting.
I20260620 04:30:46.228331 8434 tablet_bootstrap.cc:492] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: Bootstrap starting.
I20260620 04:30:46.228577 8433 tablet_bootstrap.cc:654] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.230016 8433 log.cc:824] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:46.230549 8434 tablet_bootstrap.cc:654] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.231341 8434 log.cc:824] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:46.232039 8435 tablet_bootstrap.cc:492] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: Bootstrap starting.
I20260620 04:30:46.232236 8433 tablet_bootstrap.cc:492] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: No bootstrap required, opened a new log
I20260620 04:30:46.232429 8433 ts_tablet_manager.cc:1397] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:30:46.233095 8434 tablet_bootstrap.cc:492] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: No bootstrap required, opened a new log
I20260620 04:30:46.233278 8434 ts_tablet_manager.cc:1397] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:30:46.234320 8435 tablet_bootstrap.cc:654] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.235288 8435 log.cc:824] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:46.236481 8433 raft_consensus.cc:348] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } }
I20260620 04:30:46.236735 8433 raft_consensus.cc:374] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.236796 8433 raft_consensus.cc:729] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810b9e5727f44bfc8e33ad717d1a6174, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.237270 8433 consensus_queue.cc:260] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } }
I20260620 04:30:46.237783 8435 tablet_bootstrap.cc:492] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: No bootstrap required, opened a new log
I20260620 04:30:46.237627 8434 raft_consensus.cc:348] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } }
I20260620 04:30:46.237922 8434 raft_consensus.cc:374] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.237969 8434 raft_consensus.cc:729] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee9801a81e84f21afe0f60f8e3c08b0, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.238001 8435 ts_tablet_manager.cc:1397] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:30:46.238019 8405 heartbeater.cc:503] Master 127.0.98.62:43185 was elected leader, sending a full tablet report...
I20260620 04:30:46.238430 8433 ts_tablet_manager.cc:1428] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
I20260620 04:30:46.238425 8434 consensus_queue.cc:260] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } }
I20260620 04:30:46.239156 8125 heartbeater.cc:503] Master 127.0.98.62:43185 was elected leader, sending a full tablet report...
I20260620 04:30:46.239399 8434 ts_tablet_manager.cc:1428] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260620 04:30:46.241783 8435 raft_consensus.cc:348] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } }
I20260620 04:30:46.242025 8435 raft_consensus.cc:374] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.242085 8435 raft_consensus.cc:729] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6da02a19fc4f3ea8b25e330a2bf082, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.242537 8435 consensus_queue.cc:260] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } }
I20260620 04:30:46.243211 8265 heartbeater.cc:503] Master 127.0.98.62:43185 was elected leader, sending a full tablet report...
I20260620 04:30:46.243562 8435 ts_tablet_manager.cc:1428] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:30:46.257684 8440 raft_consensus.cc:482] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:46.257874 8440 raft_consensus.cc:504] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } }
I20260620 04:30:46.258955 8440 leader_election.cc:288] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677), bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077)
I20260620 04:30:46.263026 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.259225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38677 (local address 127.0.98.1:35341)
0620 04:30:46.259552 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.259566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.259696 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.259970 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.259973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.260001 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:46.260117 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.260122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.260849 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.260853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.261818 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.261828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.262580 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.262583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.262677 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.262704 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.262773 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.262851 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":3,"thread_start_us":124,"threads_started":1}
I20260620 04:30:46.263092 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.259324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35341 (local address 127.0.98.3:38677)
0620 04:30:46.259487 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.259491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.259661 (+ 170us) server_negotiation.cc:407] Connection header received
0620 04:30:46.259816 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.259819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.259875 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.259947 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:46.260217 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.260746 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.261939 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.262457 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.262563 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.262625 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.262832 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.262933 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 04:30:46.263345 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.259767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58151 (local address 127.0.98.2:45077)
0620 04:30:46.259913 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.259918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.260082 (+ 164us) server_negotiation.cc:407] Connection header received
0620 04:30:46.260178 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.260181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.260223 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.260280 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:46.260583 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.261135 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.262224 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.262677 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.262795 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.262860 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.263091 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.263192 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 04:30:46.263458 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.259626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45077 (local address 127.0.98.1:58151)
0620 04:30:46.259991 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.260001 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.260094 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.260298 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.260300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.260314 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:46.260474 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.260479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.261232 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.261235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.262110 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.262121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.262797 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.262800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.262908 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.262929 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.263039 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.263238 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":90,"threads_started":1}
I20260620 04:30:46.263900 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80a1aa24e8534fddbe0a48bf0ddce598" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810b9e5727f44bfc8e33ad717d1a6174" is_pre_election: true
I20260620 04:30:46.264022 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80a1aa24e8534fddbe0a48bf0ddce598" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" is_pre_election: true
I20260620 04:30:46.264254 8359 raft_consensus.cc:2454] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 0.
I20260620 04:30:46.264253 8219 raft_consensus.cc:2454] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 0.
I20260620 04:30:46.264715 8012 leader_election.cc:302] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee9801a81e84f21afe0f60f8e3c08b0, bd6da02a19fc4f3ea8b25e330a2bf082; no voters:
I20260620 04:30:46.264972 8440 raft_consensus.cc:2781] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:46.265120 8440 raft_consensus.cc:482] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:46.265169 8440 raft_consensus.cc:3037] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.266268 8440 raft_consensus.cc:504] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } }
I20260620 04:30:46.266616 8440 leader_election.cc:288] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 election: Requested vote from peers 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677), bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077)
I20260620 04:30:46.266999 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80a1aa24e8534fddbe0a48bf0ddce598" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810b9e5727f44bfc8e33ad717d1a6174"
I20260620 04:30:46.266999 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80a1aa24e8534fddbe0a48bf0ddce598" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082"
I20260620 04:30:46.267130 8359 raft_consensus.cc:3037] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.267130 8219 raft_consensus.cc:3037] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.268431 8359 raft_consensus.cc:2454] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 1.
I20260620 04:30:46.268748 8219 raft_consensus.cc:2454] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 1.
I20260620 04:30:46.268752 8012 leader_election.cc:302] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee9801a81e84f21afe0f60f8e3c08b0, 810b9e5727f44bfc8e33ad717d1a6174; no voters:
I20260620 04:30:46.268913 8440 raft_consensus.cc:2781] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:46.269157 8440 raft_consensus.cc:686] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0ee9801a81e84f21afe0f60f8e3c08b0, State: Running, Role: LEADER
I20260620 04:30:46.269491 8440 consensus_queue.cc:237] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } }
I20260620 04:30:46.272758 7921 catalog_manager.cc:5495] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } health_report { overall_health: HEALTHY } } }
I20260620 04:30:46.287576 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.284048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.0.98.1:32887)
0620 04:30:46.284158 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.284161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.284173 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:30:46.284257 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.284261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.284323 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.284421 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:46.284619 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.285266 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.286133 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.286348 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.286449 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.286703 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:46.287179 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:46.287284 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.287370 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.287437 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 04:30:46.294096 8359 raft_consensus.cc:1261] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Refusing update from remote peer 0ee9801a81e84f21afe0f60f8e3c08b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:46.294096 8219 raft_consensus.cc:1261] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Refusing update from remote peer 0ee9801a81e84f21afe0f60f8e3c08b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:46.295015 8440 consensus_queue.cc:1035] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:46.295246 8444 consensus_queue.cc:1035] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:46.305166 8448 mvcc.cc:205] Tried to move back new op lower bound from 7298784650412601344 to 7298784650323038208. Current Snapshot: MvccSnapshot[applied={T|T < 7298784650412601344}]
I20260620 04:30:46.315011 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:46.315208 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:46.318027 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde: Sending DeleteTablet for 3 replicas of tablet 80a1aa24e8534fddbe0a48bf0ddce598
I20260620 04:30:46.319203 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a1aa24e8534fddbe0a48bf0ddce598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40608
I20260620 04:30:46.319207 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a1aa24e8534fddbe0a48bf0ddce598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43668
I20260620 04:30:46.319937 8458 tablet_replica.cc:330] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: stopping tablet replica
I20260620 04:30:46.320240 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a1aa24e8534fddbe0a48bf0ddce598 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35070
I20260620 04:30:46.320297 8458 raft_consensus.cc:2229] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:46.319938 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 04:30:46.320652 8458 raft_consensus.cc:2258] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 04:30:46.320851 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:46.321053 8457 tablet_replica.cc:330] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: stopping tablet replica
I20260620 04:30:46.321363 8457 raft_consensus.cc:2229] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:46.321729 8457 raft_consensus.cc:2258] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:46.323256 8457 ts_tablet_manager.cc:1905] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:46.321784 8459 tablet_replica.cc:330] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: stopping tablet replica
I20260620 04:30:46.323751 8459 raft_consensus.cc:2229] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:46.324143 8459 raft_consensus.cc:2258] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:46.323293 8458 ts_tablet_manager.cc:1905] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:46.326074 8457 ts_tablet_manager.cc:1918] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:46.326172 8457 log.cc:1196] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/80a1aa24e8534fddbe0a48bf0ddce598
I20260620 04:30:46.326469 8457 ts_tablet_manager.cc:1939] T 80a1aa24e8534fddbe0a48bf0ddce598 P 0ee9801a81e84f21afe0f60f8e3c08b0: Deleting consensus metadata
I20260620 04:30:46.326967 8459 ts_tablet_manager.cc:1905] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:46.327914 7908 catalog_manager.cc:4841] TS 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1:32887): tablet 80a1aa24e8534fddbe0a48bf0ddce598 (table test-table [id=66ea4b6d32624150b10b978dbde41e59]) successfully deleted
I20260620 04:30:46.328784 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 9d7664daf9b84cf4823445302e5258a7 (DEFAULT_TABLE table=test-table [id=a339c0d6ce8f4118a8cc3220c6a02159]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.329093 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d7664daf9b84cf4823445302e5258a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.330386 8458 ts_tablet_manager.cc:1918] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:46.331935 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 9d7664daf9b84cf4823445302e5258a7 (DEFAULT_TABLE table=test-table [id=a339c0d6ce8f4118a8cc3220c6a02159]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.331467 8435 tablet_bootstrap.cc:492] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082: Bootstrap starting.
I20260620 04:30:46.332213 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d7664daf9b84cf4823445302e5258a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.333333 8435 tablet_bootstrap.cc:654] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.333478 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 9d7664daf9b84cf4823445302e5258a7 (DEFAULT_TABLE table=test-table [id=a339c0d6ce8f4118a8cc3220c6a02159]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.333732 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d7664daf9b84cf4823445302e5258a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.333912 8458 log.cc:1196] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/80a1aa24e8534fddbe0a48bf0ddce598
I20260620 04:30:46.334548 8435 tablet_bootstrap.cc:492] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082: No bootstrap required, opened a new log
I20260620 04:30:46.334627 8435 ts_tablet_manager.cc:1397] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:46.335083 8435 raft_consensus.cc:348] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.335222 8435 raft_consensus.cc:374] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.335258 8435 raft_consensus.cc:729] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6da02a19fc4f3ea8b25e330a2bf082, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.335392 8435 consensus_queue.cc:260] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.335662 8435 ts_tablet_manager.cc:1428] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:46.336000 8433 tablet_bootstrap.cc:492] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: Bootstrap starting.
I20260620 04:30:46.336988 8433 tablet_bootstrap.cc:654] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.337987 8441 raft_consensus.cc:482] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:46.338078 8441 raft_consensus.cc:504] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.338232 8434 tablet_bootstrap.cc:492] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0: Bootstrap starting.
I20260620 04:30:46.338836 8433 tablet_bootstrap.cc:492] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: No bootstrap required, opened a new log
I20260620 04:30:46.338922 8433 ts_tablet_manager.cc:1397] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:46.339213 8434 tablet_bootstrap.cc:654] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.339322 8441 leader_election.cc:288] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1:32887), 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677)
I20260620 04:30:46.339376 8433 raft_consensus.cc:348] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.339478 8433 raft_consensus.cc:374] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.339507 8433 raft_consensus.cc:729] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810b9e5727f44bfc8e33ad717d1a6174, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.339638 8433 consensus_queue.cc:260] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.339888 8433 ts_tablet_manager.cc:1428] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:30:46.342713 8458 ts_tablet_manager.cc:1939] T 80a1aa24e8534fddbe0a48bf0ddce598 P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting consensus metadata
I20260620 04:30:46.344588 7908 catalog_manager.cc:4841] TS bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077): tablet 80a1aa24e8534fddbe0a48bf0ddce598 (table test-table [id=66ea4b6d32624150b10b978dbde41e59]) successfully deleted
I20260620 04:30:46.344817 8459 ts_tablet_manager.cc:1918] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:46.344923 8459 log.cc:1196] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/80a1aa24e8534fddbe0a48bf0ddce598
I20260620 04:30:46.345235 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.339777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32887 (local address 127.0.98.2:59749)
0620 04:30:46.339931 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.339947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.340026 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.340949 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.340952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.340973 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:46.341128 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.341135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.342147 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.342150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.343464 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.343476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.344770 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.344774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.344892 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.344916 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.344985 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.345087 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 04:30:46.345275 8459 ts_tablet_manager.cc:1939] T 80a1aa24e8534fddbe0a48bf0ddce598 P 810b9e5727f44bfc8e33ad717d1a6174: Deleting consensus metadata
I20260620 04:30:46.346172 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.340417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59749 (local address 127.0.98.1:32887)
0620 04:30:46.340548 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.340552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.340565 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:46.340609 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.340612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.340654 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.340721 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:46.341249 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.342022 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.343603 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.344239 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.344344 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.344413 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.345955 (+ 1542us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.346054 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:30:46.346678 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7664daf9b84cf4823445302e5258a7" candidate_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" is_pre_election: true
W20260620 04:30:46.347510 8152 leader_election.cc:341] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1:32887): Illegal state: must be running to vote when last-logged opid is not known
I20260620 04:30:46.348116 7908 catalog_manager.cc:4841] TS 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677): tablet 80a1aa24e8534fddbe0a48bf0ddce598 (table test-table [id=66ea4b6d32624150b10b978dbde41e59]) successfully deleted
I20260620 04:30:46.349233 8434 tablet_bootstrap.cc:492] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0: No bootstrap required, opened a new log
I20260620 04:30:46.349321 8434 ts_tablet_manager.cc:1397] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260620 04:30:46.349828 8434 raft_consensus.cc:348] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.349963 8434 raft_consensus.cc:374] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.349994 8434 raft_consensus.cc:729] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee9801a81e84f21afe0f60f8e3c08b0, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.350114 8434 consensus_queue.cc:260] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.350250 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.340021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38677 (local address 127.0.98.2:49983)
0620 04:30:46.345379 (+ 5358us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.345394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.345483 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.346521 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.346524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.346545 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:30:46.346629 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.346635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.347662 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.347665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.348820 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.348831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.349848 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.349852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.349945 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.349967 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.350028 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.350107 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5302,"thread_start_us":70,"threads_started":1}
I20260620 04:30:46.350646 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.340514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49983 (local address 127.0.98.3:38677)
0620 04:30:46.340687 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.340692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.345659 (+ 4967us) server_negotiation.cc:407] Connection header received
0620 04:30:46.345707 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.345710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.345754 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.345814 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:30:46.346756 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.347532 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.348961 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.349614 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.349692 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.349757 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.350450 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.350531 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 04:30:46.350996 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7664daf9b84cf4823445302e5258a7" candidate_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810b9e5727f44bfc8e33ad717d1a6174" is_pre_election: true
I20260620 04:30:46.351123 8359 raft_consensus.cc:2454] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6da02a19fc4f3ea8b25e330a2bf082 in term 0.
I20260620 04:30:46.351153 8434 ts_tablet_manager.cc:1428] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:46.351454 8152 leader_election.cc:302] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 810b9e5727f44bfc8e33ad717d1a6174, bd6da02a19fc4f3ea8b25e330a2bf082; no voters: 0ee9801a81e84f21afe0f60f8e3c08b0
I20260620 04:30:46.351776 8441 raft_consensus.cc:2781] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:46.351837 8441 raft_consensus.cc:482] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:46.351876 8441 raft_consensus.cc:3037] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.354242 8441 raft_consensus.cc:504] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.354651 8441 leader_election.cc:288] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [CANDIDATE]: Term 1 election: Requested vote from peers 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1:32887), 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677)
I20260620 04:30:46.354986 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7664daf9b84cf4823445302e5258a7" candidate_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0"
I20260620 04:30:46.355131 8079 raft_consensus.cc:3037] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.355232 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d7664daf9b84cf4823445302e5258a7" candidate_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810b9e5727f44bfc8e33ad717d1a6174"
I20260620 04:30:46.355456 8359 raft_consensus.cc:3037] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.356508 8079 raft_consensus.cc:2454] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6da02a19fc4f3ea8b25e330a2bf082 in term 1.
I20260620 04:30:46.357228 8359 raft_consensus.cc:2454] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6da02a19fc4f3ea8b25e330a2bf082 in term 1.
I20260620 04:30:46.357858 8152 leader_election.cc:302] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 810b9e5727f44bfc8e33ad717d1a6174, bd6da02a19fc4f3ea8b25e330a2bf082; no voters:
I20260620 04:30:46.358146 8441 raft_consensus.cc:2781] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:46.358372 8441 raft_consensus.cc:686] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 LEADER]: Becoming Leader. State: Replica: bd6da02a19fc4f3ea8b25e330a2bf082, State: Running, Role: LEADER
I20260620 04:30:46.358675 8441 consensus_queue.cc:237] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.361217 7922 catalog_manager.cc:5495] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } health_report { overall_health: UNKNOWN } } }
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
W20260620 04:30:46.398016 8406 tablet.cc:2369] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:30:46.400468 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.391044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43185 (local address 127.0.0.1:54246)
0620 04:30:46.391360 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.391377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.391491 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.391794 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.391797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.392270 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:46.392373 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.392379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.393468 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.393471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.394054 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.394062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.394642 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.394644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.394715 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.395376 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:46.395389 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:46.396859 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:46.398794 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:46.398802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:46.398807 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:46.399058 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:46.399394 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:46.399397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:46.399398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:46.399491 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:46.399934 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:46.399941 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:46.400077 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.400176 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.400322 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":112,"threads_started":1}
I20260620 04:30:46.400592 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.391086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.98.62:43185)
0620 04:30:46.391218 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.391223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.391440 (+ 217us) server_negotiation.cc:407] Connection header received
0620 04:30:46.391585 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.391588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.391638 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.391729 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:46.392509 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.393352 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.394162 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.394505 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.394611 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.397045 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:46.397050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:46.397053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:46.397081 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:46.398665 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:46.399145 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:46.399149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:46.399151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:46.399281 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:46.399580 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:46.399583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:46.399585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:46.399813 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:46.399911 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.400243 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.400408 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":47}
I20260620 04:30:46.405169 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54246:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 04:30:46.405973 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:46.410890 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 697788e80306445b9cd2e7bf3b90aba0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=472e39e22a7c47a2bd419f5bcce77999]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:30:46.410916 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 697788e80306445b9cd2e7bf3b90aba0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=472e39e22a7c47a2bd419f5bcce77999]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:30:46.411139 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 697788e80306445b9cd2e7bf3b90aba0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=472e39e22a7c47a2bd419f5bcce77999]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 04:30:46.411329 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697788e80306445b9cd2e7bf3b90aba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.411342 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697788e80306445b9cd2e7bf3b90aba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.411461 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697788e80306445b9cd2e7bf3b90aba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.414245 8435 tablet_bootstrap.cc:492] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: Bootstrap starting.
I20260620 04:30:46.415186 8435 tablet_bootstrap.cc:654] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.415804 8433 tablet_bootstrap.cc:492] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: Bootstrap starting.
I20260620 04:30:46.416749 8433 tablet_bootstrap.cc:654] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.417905 8433 tablet_bootstrap.cc:492] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: No bootstrap required, opened a new log
I20260620 04:30:46.417994 8433 ts_tablet_manager.cc:1397] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:46.418398 8433 raft_consensus.cc:348] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.418519 8433 raft_consensus.cc:374] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.418550 8433 raft_consensus.cc:729] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810b9e5727f44bfc8e33ad717d1a6174, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.418570 8435 tablet_bootstrap.cc:492] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: No bootstrap required, opened a new log
I20260620 04:30:46.418646 8435 ts_tablet_manager.cc:1397] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 04:30:46.418674 8433 consensus_queue.cc:260] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.418907 8433 ts_tablet_manager.cc:1428] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:46.419067 8435 raft_consensus.cc:348] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.419180 8435 raft_consensus.cc:374] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.419209 8435 raft_consensus.cc:729] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6da02a19fc4f3ea8b25e330a2bf082, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.419155 8439 tablet_replica.cc:439] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } }
I20260620 04:30:46.419294 8439 tablet_replica.cc:442] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:46.419319 8435 consensus_queue.cc:260] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.419571 8435 ts_tablet_manager.cc:1428] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:30:46.419545 8441 tablet_replica.cc:439] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } }
I20260620 04:30:46.419672 8441 tablet_replica.cc:442] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:46.421232 8434 tablet_bootstrap.cc:492] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Bootstrap starting.
I20260620 04:30:46.422138 8434 tablet_bootstrap.cc:654] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.423434 8434 tablet_bootstrap.cc:492] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: No bootstrap required, opened a new log
I20260620 04:30:46.423527 8434 ts_tablet_manager.cc:1397] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:46.423947 8434 raft_consensus.cc:348] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.424070 8434 raft_consensus.cc:374] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.424103 8434 raft_consensus.cc:729] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee9801a81e84f21afe0f60f8e3c08b0, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.424222 8434 consensus_queue.cc:260] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.424491 8434 ts_tablet_manager.cc:1428] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:46.424520 8452 tablet_replica.cc:439] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } }
I20260620 04:30:46.424696 8452 tablet_replica.cc:442] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 04:30:46.430387 8266 tablet.cc:2369] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:30:46.448334 8126 tablet.cc:2369] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:30:46.522219 8452 raft_consensus.cc:482] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:46.522384 8452 raft_consensus.cc:504] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.522859 8452 leader_election.cc:288] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077), 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677)
I20260620 04:30:46.523391 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697788e80306445b9cd2e7bf3b90aba0" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810b9e5727f44bfc8e33ad717d1a6174" is_pre_election: true
I20260620 04:30:46.523406 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697788e80306445b9cd2e7bf3b90aba0" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" is_pre_election: true
I20260620 04:30:46.523548 8219 raft_consensus.cc:2454] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 0.
I20260620 04:30:46.523548 8359 raft_consensus.cc:2454] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 0.
I20260620 04:30:46.523907 8012 leader_election.cc:302] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee9801a81e84f21afe0f60f8e3c08b0, 810b9e5727f44bfc8e33ad717d1a6174; no voters:
I20260620 04:30:46.524091 8452 raft_consensus.cc:2781] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:46.524148 8452 raft_consensus.cc:482] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:46.524202 8452 raft_consensus.cc:3037] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.525200 8452 raft_consensus.cc:504] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.525520 8452 leader_election.cc:288] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 election: Requested vote from peers bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077), 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677)
I20260620 04:30:46.525920 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697788e80306445b9cd2e7bf3b90aba0" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082"
I20260620 04:30:46.525980 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697788e80306445b9cd2e7bf3b90aba0" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810b9e5727f44bfc8e33ad717d1a6174"
I20260620 04:30:46.526058 8219 raft_consensus.cc:3037] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.526093 8359 raft_consensus.cc:3037] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.526966 8359 raft_consensus.cc:2454] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 1.
I20260620 04:30:46.527009 8219 raft_consensus.cc:2454] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 1.
I20260620 04:30:46.527258 8012 leader_election.cc:302] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee9801a81e84f21afe0f60f8e3c08b0, 810b9e5727f44bfc8e33ad717d1a6174; no voters:
I20260620 04:30:46.527438 8452 raft_consensus.cc:2781] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:46.527523 8452 raft_consensus.cc:686] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0ee9801a81e84f21afe0f60f8e3c08b0, State: Running, Role: LEADER
I20260620 04:30:46.527666 8452 consensus_queue.cc:237] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.528388 8444 tablet_replica.cc:439] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: TxnStatusTablet state changed. Reason: New leader 0ee9801a81e84f21afe0f60f8e3c08b0. Latest consensus state: current_term: 1 leader_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } }
I20260620 04:30:46.528492 8444 tablet_replica.cc:442] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:30:46.528759 8476 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 04:30:46.528931 8476 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 04:30:46.529467 7920 catalog_manager.cc:5495] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } health_report { overall_health: UNKNOWN } } }
I20260620 04:30:46.553319 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.549803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.0.98.1:32887)
0620 04:30:46.549937 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.549942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.549959 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:30:46.550064 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.550067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.550119 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.550205 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:46.550400 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.551203 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.552179 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.552412 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.552516 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.552789 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:46.552870 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:46.552928 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.553073 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.553129 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:30:46.553293 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.549692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32887 (local address 127.0.0.1:40636)
0620 04:30:46.549851 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.549866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.549959 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.550217 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.550219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.550224 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:46.550318 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.550322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.551326 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.551330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.552035 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.552046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.552521 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.552524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.552608 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.552657 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:46.552942 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:46.552958 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.553060 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.553152 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 04:30:46.556124 8359 raft_consensus.cc:1261] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Refusing update from remote peer 0ee9801a81e84f21afe0f60f8e3c08b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:46.556192 8219 raft_consensus.cc:1261] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Refusing update from remote peer 0ee9801a81e84f21afe0f60f8e3c08b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:46.556628 8444 consensus_queue.cc:1035] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:46.556840 8452 consensus_queue.cc:1035] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:46.558250 8439 tablet_replica.cc:439] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: TxnStatusTablet state changed. Reason: New leader 0ee9801a81e84f21afe0f60f8e3c08b0. Latest consensus state: current_term: 1 leader_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } }
I20260620 04:30:46.558285 8441 tablet_replica.cc:439] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: TxnStatusTablet state changed. Reason: New leader 0ee9801a81e84f21afe0f60f8e3c08b0. Latest consensus state: current_term: 1 leader_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } }
I20260620 04:30:46.558377 8439 tablet_replica.cc:442] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:46.558398 8441 tablet_replica.cc:442] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:46.560866 8444 tablet_replica.cc:439] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } }
I20260620 04:30:46.561056 8444 tablet_replica.cc:442] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:30:46.561714 8452 tablet_replica.cc:439] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } }
I20260620 04:30:46.561722 8439 tablet_replica.cc:439] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } }
I20260620 04:30:46.561828 8452 tablet_replica.cc:442] T 697788e80306445b9cd2e7bf3b90aba0 P 0ee9801a81e84f21afe0f60f8e3c08b0: This TxnStatusTablet replica's current role is: LEADER
I20260620 04:30:46.561837 8439 tablet_replica.cc:442] T 697788e80306445b9cd2e7bf3b90aba0 P 810b9e5727f44bfc8e33ad717d1a6174: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:46.562417 8441 tablet_replica.cc:439] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } } }
I20260620 04:30:46.562521 8441 tablet_replica.cc:442] T 697788e80306445b9cd2e7bf3b90aba0 P bd6da02a19fc4f3ea8b25e330a2bf082: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 04:30:46.562608 8450 mvcc.cc:205] Tried to move back new op lower bound from 7298784651490148352 to 7298784651379306496. Current Snapshot: MvccSnapshot[applied={T|T < 7298784651490148352 or (T in {7298784651490148352})}]
I20260620 04:30:46.570663 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.566582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.0.98.2:45077)
0620 04:30:46.566692 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.566697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.566711 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:46.566823 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.566826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.566893 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.567004 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:46.567315 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.568091 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.568940 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.569319 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.569431 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.569640 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:46.570230 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:46.570339 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.570434 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.570494 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:30:46.581696 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.578346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32887 (local address 127.0.98.2:56205)
0620 04:30:46.578651 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.578665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.578778 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.579023 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.579025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.579033 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:46.579149 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.579154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.579898 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.579901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.580522 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.580530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.580964 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.580967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.581093 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.581144 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:46.581415 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:46.581421 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.581478 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.581556 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":105,"threads_started":1}
I20260620 04:30:46.581682 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.578490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56205 (local address 127.0.98.1:32887)
0620 04:30:46.578630 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.578634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.578743 (+ 109us) server_negotiation.cc:407] Connection header received
0620 04:30:46.578879 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.578882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.578941 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.579003 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:46.579286 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.579795 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.580629 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.580864 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.580949 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.581272 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:46.581344 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:46.581401 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.581488 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.581540 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 04:30:46.588635 8079 raft_consensus.cc:1261] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Refusing update from remote peer bd6da02a19fc4f3ea8b25e330a2bf082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:46.588871 8359 raft_consensus.cc:1261] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Refusing update from remote peer bd6da02a19fc4f3ea8b25e330a2bf082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:46.589172 8441 consensus_queue.cc:1035] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:46.589414 8464 consensus_queue.cc:1035] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:46.596838 8447 mvcc.cc:205] Tried to move back new op lower bound from 7298784651622891520 to 7298784650689253376. Current Snapshot: MvccSnapshot[applied={T|T < 7298784651622891520}]
I20260620 04:30:46.617311 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.609555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39387 (local address 127.0.98.2:45077)
0620 04:30:46.609714 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.609719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.613323 (+ 3604us) server_negotiation.cc:407] Connection header received
0620 04:30:46.613480 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.613483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.613544 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.613616 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:46.613904 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.614669 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.615949 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.616209 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.616323 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.616748 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:46.616871 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:46.616944 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.617101 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.617156 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 04:30:46.617362 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.609489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45077 (local address 127.0.98.1:39387)
0620 04:30:46.613239 (+ 3750us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:46.613254 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:46.613369 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:46.613635 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:46.613638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:46.613648 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:46.613762 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.613768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.614808 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.614811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.615796 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:46.615806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.616454 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.616457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:46.616548 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.616608 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:46.616961 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:46.616968 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 04:30:46.617083 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 04:30:46.617171 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3614,"thread_start_us":104,"threads_started":1}
I20260620 04:30:46.642266 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:46.642449 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:46.644964 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde: Sending DeleteTablet for 3 replicas of tablet 9d7664daf9b84cf4823445302e5258a7
I20260620 04:30:46.645758 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7664daf9b84cf4823445302e5258a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35070
I20260620 04:30:46.645758 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7664daf9b84cf4823445302e5258a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43668
I20260620 04:30:46.645962 8458 tablet_replica.cc:330] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082: stopping tablet replica
I20260620 04:30:46.646071 8458 raft_consensus.cc:2229] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:46.646381 8458 raft_consensus.cc:2258] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:46.646579 8459 tablet_replica.cc:330] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: stopping tablet replica
I20260620 04:30:46.646710 8459 raft_consensus.cc:2229] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:46.646821 8459 raft_consensus.cc:2258] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:46.647270 8458 ts_tablet_manager.cc:1905] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:46.647990 8459 ts_tablet_manager.cc:1905] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:46.650393 8458 ts_tablet_manager.cc:1918] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:46.650480 8458 log.cc:1196] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/9d7664daf9b84cf4823445302e5258a7
I20260620 04:30:46.650960 8458 ts_tablet_manager.cc:1939] T 9d7664daf9b84cf4823445302e5258a7 P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting consensus metadata
I20260620 04:30:46.651806 7908 catalog_manager.cc:4841] TS bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077): tablet 9d7664daf9b84cf4823445302e5258a7 (table test-table [id=a339c0d6ce8f4118a8cc3220c6a02159]) successfully deleted
I20260620 04:30:46.655251 8459 ts_tablet_manager.cc:1918] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:46.655359 8459 log.cc:1196] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/9d7664daf9b84cf4823445302e5258a7
I20260620 04:30:46.655593 8459 ts_tablet_manager.cc:1939] T 9d7664daf9b84cf4823445302e5258a7 P 810b9e5727f44bfc8e33ad717d1a6174: Deleting consensus metadata
I20260620 04:30:46.656345 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d7664daf9b84cf4823445302e5258a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40608
I20260620 04:30:46.656488 7908 catalog_manager.cc:4841] TS 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677): tablet 9d7664daf9b84cf4823445302e5258a7 (table test-table [id=a339c0d6ce8f4118a8cc3220c6a02159]) successfully deleted
I20260620 04:30:46.656744 8457 tablet_replica.cc:330] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0: stopping tablet replica
I20260620 04:30:46.656909 8457 raft_consensus.cc:2229] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:46.657066 8457 raft_consensus.cc:2258] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:46.657979 8457 ts_tablet_manager.cc:1905] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:46.661976 8457 ts_tablet_manager.cc:1918] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:46.662151 8457 log.cc:1196] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/9d7664daf9b84cf4823445302e5258a7
I20260620 04:30:46.662433 8457 ts_tablet_manager.cc:1939] T 9d7664daf9b84cf4823445302e5258a7 P 0ee9801a81e84f21afe0f60f8e3c08b0: Deleting consensus metadata
I20260620 04:30:46.663280 7908 catalog_manager.cc:4841] TS 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1:32887): tablet 9d7664daf9b84cf4823445302e5258a7 (table test-table [id=a339c0d6ce8f4118a8cc3220c6a02159]) successfully deleted
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:30:46.689329 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:46.681690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.0.98.62:43185)
0620 04:30:46.681815 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.681820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.681972 (+ 152us) server_negotiation.cc:407] Connection header received
0620 04:30:46.682153 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.682156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.682212 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.682295 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:46.683170 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.684028 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.685313 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.685605 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.685704 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.689208 (+ 3504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54248: BlockingRecv error: recv got EOF from 127.0.0.1:54248 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 04:30:46 dist-test-slave-rzqm krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 04:30:46.700763 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.692246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.0.98.62:43185)
0620 04:30:46.692391 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.692395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.692584 (+ 189us) server_negotiation.cc:407] Connection header received
0620 04:30:46.692738 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.692741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.692792 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.692870 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:46.693635 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.694160 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.695194 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.695466 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.695558 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.697855 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:46.697859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:46.697862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:46.697890 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:46.699199 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:46.699634 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:46.699637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:46.699639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:46.699721 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:46.699961 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:46.699965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:46.699967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:46.700150 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:46.700259 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.700489 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.700600 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 04:30:46.703476 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:46.704041 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:46.709450 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 938a1dc2f11d4d11aa0687887bee9fd0 (DEFAULT_TABLE table=test-table [id=ffa61e19f8ec47aaa0d1840a20bc16b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.709450 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 938a1dc2f11d4d11aa0687887bee9fd0 (DEFAULT_TABLE table=test-table [id=ffa61e19f8ec47aaa0d1840a20bc16b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.709766 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938a1dc2f11d4d11aa0687887bee9fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.709767 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938a1dc2f11d4d11aa0687887bee9fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.712183 8433 tablet_bootstrap.cc:492] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174: Bootstrap starting.
I20260620 04:30:46.712240 8435 tablet_bootstrap.cc:492] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082: Bootstrap starting.
I20260620 04:30:46.713079 8435 tablet_bootstrap.cc:654] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.713218 8433 tablet_bootstrap.cc:654] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.713999 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 938a1dc2f11d4d11aa0687887bee9fd0 (DEFAULT_TABLE table=test-table [id=ffa61e19f8ec47aaa0d1840a20bc16b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.714277 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938a1dc2f11d4d11aa0687887bee9fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.716687 8434 tablet_bootstrap.cc:492] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Bootstrap starting.
I20260620 04:30:46.717549 8434 tablet_bootstrap.cc:654] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.718767 8434 tablet_bootstrap.cc:492] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0: No bootstrap required, opened a new log
I20260620 04:30:46.718854 8434 ts_tablet_manager.cc:1397] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:46.719211 8434 raft_consensus.cc:348] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.719316 8434 raft_consensus.cc:374] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.719352 8434 raft_consensus.cc:729] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee9801a81e84f21afe0f60f8e3c08b0, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.719466 8434 consensus_queue.cc:260] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.719702 8434 ts_tablet_manager.cc:1428] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:30:46.725463 8433 tablet_bootstrap.cc:492] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174: No bootstrap required, opened a new log
I20260620 04:30:46.725584 8433 ts_tablet_manager.cc:1397] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20260620 04:30:46.725719 8435 tablet_bootstrap.cc:492] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082: No bootstrap required, opened a new log
I20260620 04:30:46.725808 8435 ts_tablet_manager.cc:1397] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.001s
I20260620 04:30:46.726114 8433 raft_consensus.cc:348] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.726238 8433 raft_consensus.cc:374] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.726274 8433 raft_consensus.cc:729] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810b9e5727f44bfc8e33ad717d1a6174, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.726244 8435 raft_consensus.cc:348] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.726344 8435 raft_consensus.cc:374] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.726372 8435 raft_consensus.cc:729] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6da02a19fc4f3ea8b25e330a2bf082, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.726404 8433 consensus_queue.cc:260] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.726475 8435 consensus_queue.cc:260] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.726670 8433 ts_tablet_manager.cc:1428] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:30:46.726720 8435 ts_tablet_manager.cc:1428] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 04:30:46.925853 8485 raft_consensus.cc:482] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:46.926008 8485 raft_consensus.cc:504] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.926441 8485 leader_election.cc:288] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1:32887), 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677)
I20260620 04:30:46.926998 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938a1dc2f11d4d11aa0687887bee9fd0" candidate_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810b9e5727f44bfc8e33ad717d1a6174" is_pre_election: true
I20260620 04:30:46.927000 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938a1dc2f11d4d11aa0687887bee9fd0" candidate_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" is_pre_election: true
I20260620 04:30:46.927155 8359 raft_consensus.cc:2454] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6da02a19fc4f3ea8b25e330a2bf082 in term 0.
I20260620 04:30:46.927196 8079 raft_consensus.cc:2454] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd6da02a19fc4f3ea8b25e330a2bf082 in term 0.
I20260620 04:30:46.927487 8152 leader_election.cc:302] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 810b9e5727f44bfc8e33ad717d1a6174, bd6da02a19fc4f3ea8b25e330a2bf082; no voters:
I20260620 04:30:46.927673 8485 raft_consensus.cc:2781] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:46.927752 8485 raft_consensus.cc:482] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:46.927798 8485 raft_consensus.cc:3037] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.929085 8485 raft_consensus.cc:504] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.929432 8485 leader_election.cc:288] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [CANDIDATE]: Term 1 election: Requested vote from peers 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1:32887), 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677)
I20260620 04:30:46.929817 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938a1dc2f11d4d11aa0687887bee9fd0" candidate_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0"
I20260620 04:30:46.929873 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938a1dc2f11d4d11aa0687887bee9fd0" candidate_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810b9e5727f44bfc8e33ad717d1a6174"
I20260620 04:30:46.929949 8079 raft_consensus.cc:3037] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.929999 8359 raft_consensus.cc:3037] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.930924 8079 raft_consensus.cc:2454] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6da02a19fc4f3ea8b25e330a2bf082 in term 1.
I20260620 04:30:46.931125 8359 raft_consensus.cc:2454] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd6da02a19fc4f3ea8b25e330a2bf082 in term 1.
I20260620 04:30:46.931219 8152 leader_election.cc:302] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee9801a81e84f21afe0f60f8e3c08b0, bd6da02a19fc4f3ea8b25e330a2bf082; no voters:
I20260620 04:30:46.931401 8485 raft_consensus.cc:2781] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:46.931496 8485 raft_consensus.cc:686] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 LEADER]: Becoming Leader. State: Replica: bd6da02a19fc4f3ea8b25e330a2bf082, State: Running, Role: LEADER
I20260620 04:30:46.931634 8485 consensus_queue.cc:237] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.933465 7920 catalog_manager.cc:5495] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } health_report { overall_health: UNKNOWN } } }
I20260620 04:30:46.959156 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:46.956157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.0.98.2:45077)
0620 04:30:46.956308 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:46.956312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:46.956325 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:46.956370 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:46.956373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:46.956416 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:46.956478 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:46.956783 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.957323 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.958230 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:46.958454 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:46.958542 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:46.958779 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:46.958852 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:46.958899 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:46.958993 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 04:30:46.959036 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 04:30:46.961326 8359 raft_consensus.cc:1261] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Refusing update from remote peer bd6da02a19fc4f3ea8b25e330a2bf082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:46.961342 8079 raft_consensus.cc:1261] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Refusing update from remote peer bd6da02a19fc4f3ea8b25e330a2bf082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:46.961802 8485 consensus_queue.cc:1035] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:46.961987 8464 consensus_queue.cc:1035] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:46.969352 7922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:46.969520 7922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:46.971684 7922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde: Sending DeleteTablet for 3 replicas of tablet 938a1dc2f11d4d11aa0687887bee9fd0
I20260620 04:30:46.972240 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet 938a1dc2f11d4d11aa0687887bee9fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40608
I20260620 04:30:46.972272 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 938a1dc2f11d4d11aa0687887bee9fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43668
I20260620 04:30:46.972321 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet 938a1dc2f11d4d11aa0687887bee9fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:46 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35070
I20260620 04:30:46.972476 8458 tablet_replica.cc:330] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082: stopping tablet replica
I20260620 04:30:46.972502 8457 tablet_replica.cc:330] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0: stopping tablet replica
I20260620 04:30:46.972589 8458 raft_consensus.cc:2229] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:46.972599 8457 raft_consensus.cc:2229] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:46.972708 8457 raft_consensus.cc:2258] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:46.972744 8458 raft_consensus.cc:2258] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:46.972851 8459 tablet_replica.cc:330] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174: stopping tablet replica
I20260620 04:30:46.972957 8459 raft_consensus.cc:2229] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:46.973114 8459 raft_consensus.cc:2258] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:46.973403 8458 ts_tablet_manager.cc:1905] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:46.974102 8459 ts_tablet_manager.cc:1905] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:46.974225 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:30:46.974695 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:30:46.976068 8458 ts_tablet_manager.cc:1918] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:46.976157 8458 log.cc:1196] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/938a1dc2f11d4d11aa0687887bee9fd0
I20260620 04:30:46.976411 8458 ts_tablet_manager.cc:1939] T 938a1dc2f11d4d11aa0687887bee9fd0 P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting consensus metadata
I20260620 04:30:46.976440 8459 ts_tablet_manager.cc:1918] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:46.976500 8459 log.cc:1196] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/938a1dc2f11d4d11aa0687887bee9fd0
I20260620 04:30:46.976711 8459 ts_tablet_manager.cc:1939] T 938a1dc2f11d4d11aa0687887bee9fd0 P 810b9e5727f44bfc8e33ad717d1a6174: Deleting consensus metadata
I20260620 04:30:46.977403 8457 ts_tablet_manager.cc:1905] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:46.977411 7908 catalog_manager.cc:4841] TS bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077): tablet 938a1dc2f11d4d11aa0687887bee9fd0 (table test-table [id=ffa61e19f8ec47aaa0d1840a20bc16b2]) successfully deleted
I20260620 04:30:46.977569 7908 catalog_manager.cc:4841] TS 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677): tablet 938a1dc2f11d4d11aa0687887bee9fd0 (table test-table [id=ffa61e19f8ec47aaa0d1840a20bc16b2]) successfully deleted
I20260620 04:30:46.979863 8457 ts_tablet_manager.cc:1918] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:30:46.979956 8457 log.cc:1196] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/938a1dc2f11d4d11aa0687887bee9fd0
I20260620 04:30:46.980186 8457 ts_tablet_manager.cc:1939] T 938a1dc2f11d4d11aa0687887bee9fd0 P 0ee9801a81e84f21afe0f60f8e3c08b0: Deleting consensus metadata
I20260620 04:30:46.981477 7908 catalog_manager.cc:4841] TS 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1:32887): tablet 938a1dc2f11d4d11aa0687887bee9fd0 (table test-table [id=ffa61e19f8ec47aaa0d1840a20bc16b2]) successfully deleted
I20260620 04:30:46.981629 8059 tablet_service.cc:1467] Processing CreateTablet for tablet a1a24952503147b7b7c74c8b3f7c03eb (DEFAULT_TABLE table=test-table [id=a972474579154b9095b6e3a162e51358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.981662 8199 tablet_service.cc:1467] Processing CreateTablet for tablet a1a24952503147b7b7c74c8b3f7c03eb (DEFAULT_TABLE table=test-table [id=a972474579154b9095b6e3a162e51358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.981817 8339 tablet_service.cc:1467] Processing CreateTablet for tablet a1a24952503147b7b7c74c8b3f7c03eb (DEFAULT_TABLE table=test-table [id=a972474579154b9095b6e3a162e51358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:30:46.981896 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a24952503147b7b7c74c8b3f7c03eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.981921 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a24952503147b7b7c74c8b3f7c03eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.982074 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a24952503147b7b7c74c8b3f7c03eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:46.983814 8434 tablet_bootstrap.cc:492] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0: Bootstrap starting.
I20260620 04:30:46.984324 8435 tablet_bootstrap.cc:492] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082: Bootstrap starting.
I20260620 04:30:46.984679 8434 tablet_bootstrap.cc:654] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.985086 8435 tablet_bootstrap.cc:654] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.985584 8433 tablet_bootstrap.cc:492] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174: Bootstrap starting.
I20260620 04:30:46.985841 8434 tablet_bootstrap.cc:492] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0: No bootstrap required, opened a new log
I20260620 04:30:46.985917 8434 ts_tablet_manager.cc:1397] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 04:30:46.986334 8433 tablet_bootstrap.cc:654] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:46.986301 8434 raft_consensus.cc:348] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.986392 8434 raft_consensus.cc:374] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.986413 8434 raft_consensus.cc:729] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee9801a81e84f21afe0f60f8e3c08b0, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.986536 8434 consensus_queue.cc:260] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.986779 8434 ts_tablet_manager.cc:1428] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:30:46.987629 8435 tablet_bootstrap.cc:492] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082: No bootstrap required, opened a new log
I20260620 04:30:46.987743 8435 ts_tablet_manager.cc:1397] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 04:30:46.988173 8435 raft_consensus.cc:348] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.988293 8435 raft_consensus.cc:374] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.988317 8435 raft_consensus.cc:729] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6da02a19fc4f3ea8b25e330a2bf082, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.988438 8435 consensus_queue.cc:260] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.988688 8435 ts_tablet_manager.cc:1428] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:30:46.989233 8433 tablet_bootstrap.cc:492] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174: No bootstrap required, opened a new log
I20260620 04:30:46.989322 8433 ts_tablet_manager.cc:1397] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 04:30:46.989751 8433 raft_consensus.cc:348] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.989862 8433 raft_consensus.cc:374] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:46.989898 8433 raft_consensus.cc:729] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810b9e5727f44bfc8e33ad717d1a6174, State: Initialized, Role: FOLLOWER
I20260620 04:30:46.990027 8433 consensus_queue.cc:260] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.990258 8433 ts_tablet_manager.cc:1428] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 04:30:46.993800 8452 raft_consensus.cc:482] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:30:46.993909 8452 raft_consensus.cc:504] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.994295 8452 leader_election.cc:288] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077), 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677)
I20260620 04:30:46.994645 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a24952503147b7b7c74c8b3f7c03eb" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810b9e5727f44bfc8e33ad717d1a6174" is_pre_election: true
I20260620 04:30:46.994678 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a24952503147b7b7c74c8b3f7c03eb" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" is_pre_election: true
I20260620 04:30:46.994766 8359 raft_consensus.cc:2454] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 0.
I20260620 04:30:46.994798 8219 raft_consensus.cc:2454] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 0.
I20260620 04:30:46.995056 8012 leader_election.cc:302] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee9801a81e84f21afe0f60f8e3c08b0, bd6da02a19fc4f3ea8b25e330a2bf082; no voters:
I20260620 04:30:46.995249 8452 raft_consensus.cc:2781] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:30:46.995302 8452 raft_consensus.cc:482] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:30:46.995328 8452 raft_consensus.cc:3037] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.996222 8452 raft_consensus.cc:504] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.996501 8452 leader_election.cc:288] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 election: Requested vote from peers bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077), 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677)
I20260620 04:30:46.996786 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a24952503147b7b7c74c8b3f7c03eb" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082"
I20260620 04:30:46.996877 8219 raft_consensus.cc:3037] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.996869 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a24952503147b7b7c74c8b3f7c03eb" candidate_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810b9e5727f44bfc8e33ad717d1a6174"
I20260620 04:30:46.996981 8359 raft_consensus.cc:3037] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:46.997768 8219 raft_consensus.cc:2454] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 1.
I20260620 04:30:46.997844 8359 raft_consensus.cc:2454] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee9801a81e84f21afe0f60f8e3c08b0 in term 1.
I20260620 04:30:46.998015 8012 leader_election.cc:302] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee9801a81e84f21afe0f60f8e3c08b0, bd6da02a19fc4f3ea8b25e330a2bf082; no voters:
I20260620 04:30:46.998183 8452 raft_consensus.cc:2781] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:46.998270 8452 raft_consensus.cc:686] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0ee9801a81e84f21afe0f60f8e3c08b0, State: Running, Role: LEADER
I20260620 04:30:46.998399 8452 consensus_queue.cc:237] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } }
I20260620 04:30:46.999790 7920 catalog_manager.cc:5495] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee9801a81e84f21afe0f60f8e3c08b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 } health_report { overall_health: UNKNOWN } } }
I20260620 04:30:47.016403 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:47.013548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.0.98.1:32887)
0620 04:30:47.013654 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:47.013659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:47.013672 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:30:47.013807 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:47.013810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:47.013850 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:47.013908 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:47.014131 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:47.014646 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:47.015439 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:47.015647 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:47.015731 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:47.015964 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:47.016034 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:47.016104 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:47.016214 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 04:30:47.016268 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 04:30:47.024675 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:47.021988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52875 (local address 127.0.98.1:32887)
0620 04:30:47.022177 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:47.022180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:47.022189 (+ 9us) server_negotiation.cc:407] Connection header received
0620 04:30:47.022218 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:47.022220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:47.022254 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:47.022310 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:30:47.022535 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:47.023020 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:47.023791 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:47.023985 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:47.024067 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:47.024273 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:30:47.024337 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:30:47.024390 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:47.024482 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 04:30:47.024524 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":129}
I20260620 04:30:47.024675 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:47.021867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32887 (local address 127.0.98.1:52875)
0620 04:30:47.022006 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:47.022019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:47.022085 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:47.022328 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:47.022331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:47.022338 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 04:30:47.022453 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:47.022458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:47.023119 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:47.023122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:47.023692 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:47.023700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:47.024075 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:47.024078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:47.024160 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:47.024181 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 04:30:47.024407 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 04:30:47.024412 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:30:47.024464 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 04:30:47.024552 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":19}
I20260620 04:30:47.029656 8359 raft_consensus.cc:1261] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Refusing update from remote peer 0ee9801a81e84f21afe0f60f8e3c08b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:47.029692 8219 raft_consensus.cc:1261] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Refusing update from remote peer 0ee9801a81e84f21afe0f60f8e3c08b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:30:47.030004 8440 consensus_queue.cc:1035] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "810b9e5727f44bfc8e33ad717d1a6174" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:47.030191 8444 consensus_queue.cc:1035] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6da02a19fc4f3ea8b25e330a2bf082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:30:47.060875 7922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:47.061057 7922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:30:47.063354 7922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451ff04375174e2bae5ca68ab318fcde: Sending DeleteTablet for 3 replicas of tablet a1a24952503147b7b7c74c8b3f7c03eb
I20260620 04:30:47.063881 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a24952503147b7b7c74c8b3f7c03eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40608
I20260620 04:30:47.063988 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a24952503147b7b7c74c8b3f7c03eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43668
I20260620 04:30:47.064088 8457 tablet_replica.cc:330] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0: stopping tablet replica
I20260620 04:30:47.064083 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a24952503147b7b7c74c8b3f7c03eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:30:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35070
I20260620 04:30:47.064203 8458 tablet_replica.cc:330] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082: stopping tablet replica
I20260620 04:30:47.064219 8457 raft_consensus.cc:2229] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:30:47.064256 8459 tablet_replica.cc:330] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174: stopping tablet replica
I20260620 04:30:47.064311 8458 raft_consensus.cc:2229] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:47.064374 8459 raft_consensus.cc:2229] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:30:47.064424 8458 raft_consensus.cc:2258] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:47.064422 8457 raft_consensus.cc:2258] T a1a24952503147b7b7c74c8b3f7c03eb P 0ee9801a81e84f21afe0f60f8e3c08b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:47.064530 8459 raft_consensus.cc:2258] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:30:47.065150 8458 ts_tablet_manager.cc:1905] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:47.065297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 7994
I20260620 04:30:47.065349 8459 ts_tablet_manager.cc:1905] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:30:47.067670 8459 ts_tablet_manager.cc:1918] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:47.067742 8459 log.cc:1196] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/a1a24952503147b7b7c74c8b3f7c03eb
I20260620 04:30:47.067956 8459 ts_tablet_manager.cc:1939] T a1a24952503147b7b7c74c8b3f7c03eb P 810b9e5727f44bfc8e33ad717d1a6174: Deleting consensus metadata
I20260620 04:30:47.067976 8458 ts_tablet_manager.cc:1918] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 04:30:47.068038 8458 log.cc:1196] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/a1a24952503147b7b7c74c8b3f7c03eb
I20260620 04:30:47.068274 8458 ts_tablet_manager.cc:1939] T a1a24952503147b7b7c74c8b3f7c03eb P bd6da02a19fc4f3ea8b25e330a2bf082: Deleting consensus metadata
I20260620 04:30:47.068661 7908 catalog_manager.cc:4841] TS 810b9e5727f44bfc8e33ad717d1a6174 (127.0.98.3:38677): tablet a1a24952503147b7b7c74c8b3f7c03eb (table test-table [id=a972474579154b9095b6e3a162e51358]) successfully deleted
I20260620 04:30:47.068856 7908 catalog_manager.cc:4841] TS bd6da02a19fc4f3ea8b25e330a2bf082 (127.0.98.2:45077): tablet a1a24952503147b7b7c74c8b3f7c03eb (table test-table [id=a972474579154b9095b6e3a162e51358]) successfully deleted
W20260620 04:30:47.073812 7908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32887 (error 108)
I20260620 04:30:47.074605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 8134
I20260620 04:30:47.076104 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:47.074540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32887 (local address 127.0.0.1:40648)
0620 04:30:47.075936 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:47.076009 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
W20260620 04:30:47.076229 7908 catalog_manager.cc:4584] TS 0ee9801a81e84f21afe0f60f8e3c08b0 (127.0.98.1:32887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1a24952503147b7b7c74c8b3f7c03eb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32887: connect: Connection refused (error 111)
I20260620 04:30:47.080677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 8274
I20260620 04:30:47.085574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 7890
2026-06-20T04:30:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2925 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:47 dist-test-slave-rzqm krb5kdc[8520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:47 dist-test-slave-rzqm krb5kdc[8520](info): set up 2 sockets
Jun 20 04:30:47 dist-test-slave-rzqm krb5kdc[8520](info): commencing operation
krb5kdc: starting...
W20260620 04:30:48.041522 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:48 dist-test-slave-rzqm krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:48.164726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37181
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:48.275563 8533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:48.275868 8533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:48.275909 8533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:48.280112 8533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:48.280191 8533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:48.280210 8533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:48.280227 8533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:48.280242 8533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:48.285873 8533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:48.287030 8533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:48.288111 8533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:48.296353 8539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:48.296418 8533 server_base.cc:1034] running on GCE node
W20260620 04:30:48.296355 8538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:48.296448 8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:48.297211 8533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:48.298298 8533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:48.299465 8533 hybrid_clock.cc:648] HybridClock initialized: now 1781929848299447 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T04:30:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1217 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:30:48 dist-test-slave-rzqm krb5kdc[8546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:30:48 dist-test-slave-rzqm krb5kdc[8546](info): set up 2 sockets
Jun 20 04:30:48 dist-test-slave-rzqm krb5kdc[8546](info): commencing operation
krb5kdc: starting...
W20260620 04:30:49.257397 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:30:49 dist-test-slave-rzqm krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:30:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:30:49.373699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35355
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:49.484401 8559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:49.484710 8559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:49.484751 8559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:49.488977 8559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:49.489073 8559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:49.489091 8559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:49.489108 8559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:49.489122 8559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:49.494678 8559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:49.495862 8559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:49.496917 8559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:49.505525 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:49.505535 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:49.505530 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:49.505694 8559 server_base.cc:1034] running on GCE node
I20260620 04:30:49.506371 8559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:49.507587 8559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:49.508778 8559 hybrid_clock.cc:648] HybridClock initialized: now 1781929849508747 us; error 44 us; skew 500 ppm
Jun 20 04:30:49 dist-test-slave-rzqm krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:49.515923 8559 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:49.517050 8559 webserver.cc:460] Webserver started at http://127.0.98.62:41089/ using document root <none> and password file <none>
I20260620 04:30:49.517669 8559 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:49.517747 8559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:49.517968 8559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:49.520141 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "7fb660a93d22592dcacbc62ecbf1d78a"
server_key_iv: "555ef903fa7e0f8033725cb5d032d8b1"
server_key_version: "encryptionkey@0"
I20260620 04:30:49.520722 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "7fb660a93d22592dcacbc62ecbf1d78a"
server_key_iv: "555ef903fa7e0f8033725cb5d032d8b1"
server_key_version: "encryptionkey@0"
I20260620 04:30:49.524631 8559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 04:30:49.527230 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:49.528671 8559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:49.528822 8559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "7fb660a93d22592dcacbc62ecbf1d78a"
server_key_iv: "555ef903fa7e0f8033725cb5d032d8b1"
server_key_version: "encryptionkey@0"
I20260620 04:30:49.528934 8559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:49.551375 8559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:49.555169 8559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:49.555361 8559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:49.562598 8559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35355
I20260620 04:30:49.562655 8626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35355 every 8 connection(s)
I20260620 04:30:49.563642 8559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:49.566587 8627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:30:49.570017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 8559
I20260620 04:30:49.570125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:30:49.570399 392 external_mini_cluster.cc:1428] Setting key 559c4a8317087307e0e1ec04e1dbfda0
I20260620 04:30:49.572535 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7: Bootstrap starting.
I20260620 04:30:49.575110 8627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7: Neither blocks nor log segments found. Creating new log.
I20260620 04:30:49.575850 8627 log.cc:824] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:49.578130 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7: No bootstrap required, opened a new log
Jun 20 04:30:49 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:49.581691 8627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260620 04:30:49.581985 8627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:30:49.582036 8627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5930e4ad0cd045b5b2edbf9bbc9c59f7, State: Initialized, Role: FOLLOWER
I20260620 04:30:49.582507 8627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260620 04:30:49.582635 8627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:49.582722 8627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:49.582824 8627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:30:49.583729 8627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260620 04:30:49.584076 8627 leader_election.cc:302] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5930e4ad0cd045b5b2edbf9bbc9c59f7; no voters:
I20260620 04:30:49.584404 8627 leader_election.cc:288] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:30:49.584507 8632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:30:49.584815 8632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 1 LEADER]: Becoming Leader. State: Replica: 5930e4ad0cd045b5b2edbf9bbc9c59f7, State: Running, Role: LEADER
I20260620 04:30:49.585232 8632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260620 04:30:49.585851 8627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:49.586088 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:49.571878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.0.98.62:35355)
0620 04:30:49.572289 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:49.572312 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:49.572349 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:30:49.573062 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:49.573084 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:49.573441 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:49.573733 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:49.574652 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:49.575742 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:49.576743 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:49.577301 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:49.577460 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:49.580286 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:49.580292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:49.580315 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:49.580344 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:49.582598 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:49.583105 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:49.583112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:49.583118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:49.583238 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:49.583503 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:49.583507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:49.583509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:49.584051 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:49.584235 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:49.584565 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 04:30:49.585103 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":122,"threads_started":1}
I20260620 04:30:49.586900 8634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } } }
I20260620 04:30:49.586889 8633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5930e4ad0cd045b5b2edbf9bbc9c59f7. Latest consensus state: current_term: 1 leader_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } } }
I20260620 04:30:49.587021 8634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:49.587021 8633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:49.588451 8641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:49.591084 8641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:49.597123 8641 catalog_manager.cc:1347] Generated new cluster ID: eb484e4555d54eca9cab6de82807db1b
I20260620 04:30:49.597200 8641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:49.603327 8641 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:30:49.604350 8641 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:49.609711 8641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7: Generated new TSK 0
I20260620 04:30:49.610613 8641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:30:49.642122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35355
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:49.754474 8653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:49.754770 8653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:49.754832 8653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:49.759079 8653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:49.759161 8653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:49.759244 8653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:49.764640 8653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:49.765795 8653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:49.766872 8653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:49.776641 8661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:49.776751 8659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:49.776700 8653 server_base.cc:1034] running on GCE node
W20260620 04:30:49.776713 8658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:49.777288 8653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:49.777973 8653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:49.779155 8653 hybrid_clock.cc:648] HybridClock initialized: now 1781929849779124 us; error 53 us; skew 500 ppm
Jun 20 04:30:49 dist-test-slave-rzqm krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:49.782045 8653 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:30:49.782931 8653 webserver.cc:460] Webserver started at http://127.0.98.1:41301/ using document root <none> and password file <none>
I20260620 04:30:49.783452 8653 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:49.783510 8653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:49.783681 8653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:49.785878 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "a383aab570d243739de4ef93cdbeeeb9"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "26955fb968cbbd156e9b8c687be3ae5e"
server_key_iv: "e5aa5f915f06181a6642e8f2fb8d1838"
server_key_version: "encryptionkey@0"
I20260620 04:30:49.786469 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a383aab570d243739de4ef93cdbeeeb9"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "26955fb968cbbd156e9b8c687be3ae5e"
server_key_iv: "e5aa5f915f06181a6642e8f2fb8d1838"
server_key_version: "encryptionkey@0"
I20260620 04:30:49.790338 8653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:30:49.792968 8668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:49.794274 8653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:30:49.794391 8653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "a383aab570d243739de4ef93cdbeeeb9"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "26955fb968cbbd156e9b8c687be3ae5e"
server_key_iv: "e5aa5f915f06181a6642e8f2fb8d1838"
server_key_version: "encryptionkey@0"
I20260620 04:30:49.794500 8653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:49.807859 8653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:49.811199 8653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:49.811398 8653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:49.812117 8653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:49.812969 8653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:49.813041 8653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:49.813097 8653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:49.813126 8653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:49.821556 8653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33719
I20260620 04:30:49.821579 8781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33719 every 8 connection(s)
I20260620 04:30:49.822471 8653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:49.829038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 8653
I20260620 04:30:49.829173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:30:49.829440 392 external_mini_cluster.cc:1428] Setting key 0cbf759342e1973f44b1a64251c98474
Jun 20 04:30:49 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:49.836546 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:49.824305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39963 (local address 127.0.98.62:35355)
0620 04:30:49.824463 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:49.824468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:49.825313 (+ 845us) server_negotiation.cc:407] Connection header received
0620 04:30:49.826691 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:49.826696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:49.826754 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:49.826841 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:49.828282 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:49.828838 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:49.830168 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:49.830400 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:49.830480 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:49.833132 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:49.833136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:49.833139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:49.833163 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:49.834499 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:49.834990 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:49.834994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:49.834995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:49.835069 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:49.835385 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:49.835388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:49.835390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:49.835602 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:49.835714 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:49.836235 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 04:30:49.836375 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:30:49.837440 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:49.824688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:39963)
0620 04:30:49.825161 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:49.825205 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:49.826478 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:49.827054 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:49.827066 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:49.827592 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:49.828131 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:49.828146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:49.828960 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:49.828962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:49.830066 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:49.830076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:49.830547 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:49.830552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:49.830754 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:49.831370 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:49.831385 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:49.832951 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:49.834601 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:49.834607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:49.834625 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:49.834888 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:49.835148 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:49.835151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:49.835153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:49.835292 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:49.835726 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:49.835733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:49.836002 (+ 269us) client_negotiation.cc:769] Sending connection context
0620 04:30:49.836167 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 04:30:49.836512 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:30:49.838673 8782 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35355
I20260620 04:30:49.838900 8782 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:49.839411 8782 heartbeater.cc:511] Master 127.0.98.62:35355 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:30:49.841077 8591 ts_manager.cc:194] Registered new tserver with Master: a383aab570d243739de4ef93cdbeeeb9 (127.0.98.1:33719)
I20260620 04:30:49.842448 8591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:30:49.855538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35355
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:49.966130 8787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:49.966410 8787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:49.966452 8787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:49.970597 8787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:49.970674 8787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:49.970741 8787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:49.976418 8787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:49.977572 8787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:49.978571 8787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:30:49.988629 8787 server_base.cc:1034] running on GCE node
W20260620 04:30:49.988602 8792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:49.988844 8795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:49.988595 8793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:49.989360 8787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:49.989977 8787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:49.991134 8787 hybrid_clock.cc:648] HybridClock initialized: now 1781929849991122 us; error 41 us; skew 500 ppm
Jun 20 04:30:49 dist-test-slave-rzqm krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:49.993949 8787 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:30:49.994890 8787 webserver.cc:460] Webserver started at http://127.0.98.2:43329/ using document root <none> and password file <none>
I20260620 04:30:49.995474 8787 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:49.995545 8787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:49.995749 8787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:49.997995 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "58183efd1d114742a84b82cbbc65c8dc"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "044c09dd7062ab1cdb012bfe4f478d66"
server_key_iv: "a57db82894d74d28ab1cd1992776b7ba"
server_key_version: "encryptionkey@0"
I20260620 04:30:49.998610 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "58183efd1d114742a84b82cbbc65c8dc"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "044c09dd7062ab1cdb012bfe4f478d66"
server_key_iv: "a57db82894d74d28ab1cd1992776b7ba"
server_key_version: "encryptionkey@0"
I20260620 04:30:50.002681 8787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:30:50.005410 8802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.006915 8787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:30:50.007050 8787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "58183efd1d114742a84b82cbbc65c8dc"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "044c09dd7062ab1cdb012bfe4f478d66"
server_key_iv: "a57db82894d74d28ab1cd1992776b7ba"
server_key_version: "encryptionkey@0"
I20260620 04:30:50.007169 8787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:50.020347 8787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:50.023978 8787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:50.024178 8787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:50.024829 8787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:50.025741 8787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:50.025801 8787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.025858 8787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:50.025883 8787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.034463 8787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43091
I20260620 04:30:50.034512 8915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43091 every 8 connection(s)
I20260620 04:30:50.035387 8787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:50.042094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 8787
I20260620 04:30:50.042232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:30:50.042490 392 external_mini_cluster.cc:1428] Setting key 2e6623f75a488136f12b01d4656da74c
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:50.050521 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:50.037556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57949 (local address 127.0.98.62:35355)
0620 04:30:50.037731 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:50.037735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:50.038624 (+ 889us) server_negotiation.cc:407] Connection header received
0620 04:30:50.040066 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:50.040070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:50.040127 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:50.040204 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:50.041533 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.042054 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.043425 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.043660 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.043804 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.046601 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:50.046606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:50.046608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:50.046636 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:50.048217 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:50.048849 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:50.048853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:50.048855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:50.048956 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:50.049324 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:50.049327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:50.049328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:50.049567 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:50.049670 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:50.050194 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 04:30:50.050342 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 04:30:50.051343 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:50.037947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:57949)
0620 04:30:50.038447 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:50.038494 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:50.039829 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:50.040399 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:50.040408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:50.040810 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:50.041375 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.041390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.042173 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.042177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.043299 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.043311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.043883 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.043888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.044105 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.044740 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:50.044757 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:50.046403 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:50.048380 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:50.048386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:50.048400 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:50.048744 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:50.049077 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:50.049080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:50.049082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:50.049236 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:50.049678 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:50.049683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:50.049947 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 04:30:50.050119 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 04:30:50.050406 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":144,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:30:50.052572 8916 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35355
I20260620 04:30:50.052831 8916 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:50.053438 8916 heartbeater.cc:511] Master 127.0.98.62:35355 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:30:50.054646 8591 ts_manager.cc:194] Registered new tserver with Master: 58183efd1d114742a84b82cbbc65c8dc (127.0.98.2:43091)
I20260620 04:30:50.055289 8591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:30:50.069738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35355
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:50.183776 8921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:50.184077 8921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:50.184119 8921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:50.188463 8921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:50.188546 8921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:50.188608 8921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:50.194243 8921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:50.195429 8921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:50.196425 8921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:50.205979 8929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:50.205982 8926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:50.206023 8927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:50.206338 8921 server_base.cc:1034] running on GCE node
I20260620 04:30:50.206780 8921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:50.207433 8921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:50.208599 8921 hybrid_clock.cc:648] HybridClock initialized: now 1781929850208576 us; error 38 us; skew 500 ppm
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:50.211476 8921 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:30:50.212407 8921 webserver.cc:460] Webserver started at http://127.0.98.3:41629/ using document root <none> and password file <none>
I20260620 04:30:50.212934 8921 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:50.212991 8921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:50.213212 8921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:30:50.215253 8921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "70c40e539ad4421e8d1e7c52b5a6c990"
format_stamp: "Formatted at 2026-06-20 04:30:50 on dist-test-slave-rzqm"
server_key: "49ecd4aeaf581929a315871ad9360bcb"
server_key_iv: "b203d8116da20846cf58825f0805926e"
server_key_version: "encryptionkey@0"
I20260620 04:30:50.215790 8921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "70c40e539ad4421e8d1e7c52b5a6c990"
format_stamp: "Formatted at 2026-06-20 04:30:50 on dist-test-slave-rzqm"
server_key: "49ecd4aeaf581929a315871ad9360bcb"
server_key_iv: "b203d8116da20846cf58825f0805926e"
server_key_version: "encryptionkey@0"
I20260620 04:30:50.219700 8921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 04:30:50.222414 8936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.223877 8921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:30:50.223999 8921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "70c40e539ad4421e8d1e7c52b5a6c990"
format_stamp: "Formatted at 2026-06-20 04:30:50 on dist-test-slave-rzqm"
server_key: "49ecd4aeaf581929a315871ad9360bcb"
server_key_iv: "b203d8116da20846cf58825f0805926e"
server_key_version: "encryptionkey@0"
I20260620 04:30:50.224128 8921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:50.244810 8921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:50.248138 8921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:50.248332 8921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:50.248975 8921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:50.249891 8921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:50.249938 8921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.249979 8921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:50.250003 8921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.258713 8921 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39555
I20260620 04:30:50.258738 9049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39555 every 8 connection(s)
I20260620 04:30:50.259665 8921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:50.266422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 8921
I20260620 04:30:50.266535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:30:50.266834 392 external_mini_cluster.cc:1428] Setting key 63c6fe8485723303893fad30f31c21e1
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:30:50.273499 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:50.261644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45377 (local address 127.0.98.62:35355)
0620 04:30:50.261798 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:50.261802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:50.262680 (+ 878us) server_negotiation.cc:407] Connection header received
0620 04:30:50.264094 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:50.264099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:50.264158 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:50.264235 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:50.265547 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.266080 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.267340 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.267599 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.267696 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.270231 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:50.270234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:50.270236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:50.270261 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:50.271470 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:50.271952 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:50.271956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:50.271958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:50.272034 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:50.272343 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:50.272346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:50.272347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:50.272530 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:50.272668 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:50.273205 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 04:30:50.273336 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 04:30:50.274364 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:50.262024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:45377)
0620 04:30:50.262511 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:50.262557 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:50.263867 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:50.264435 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:50.264443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:50.264826 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:50.265392 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.265407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.266221 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.266224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.267216 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.267226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.267750 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.267754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.267925 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.268492 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:50.268513 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:50.270047 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:30:50.271578 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:50.271584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:50.271602 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:50.271860 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:50.272118 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:30:50.272122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:30:50.272124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:30:50.272259 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:30:50.272673 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:30:50.272680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:30:50.272938 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 04:30:50.273163 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 04:30:50.273508 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":128,"threads_started":1}
I20260620 04:30:50.275575 9050 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35355
I20260620 04:30:50.275851 9050 heartbeater.cc:464] Registering TS with master...
I20260620 04:30:50.276389 9050 heartbeater.cc:511] Master 127.0.98.62:35355 requested a full tablet report, sending...
I20260620 04:30:50.277571 8591 ts_manager.cc:194] Registered new tserver with Master: 70c40e539ad4421e8d1e7c52b5a6c990 (127.0.98.3:39555)
I20260620 04:30:50.278178 8591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45377
I20260620 04:30:50.280794 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 04:30:50.352322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 8653
I20260620 04:30:50.357314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 8787
I20260620 04:30:50.362655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 8921
I20260620 04:30:50.367195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 8559
I20260620 04:30:50.372394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35355
--webserver_interface=127.0.98.62
--webserver_port=41089
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:30:50.482270 9059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:50.482580 9059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:50.482623 9059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:50.486820 9059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:30:50.486898 9059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:50.486917 9059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:50.486933 9059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:30:50.486948 9059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:30:50.492727 9059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:50.493954 9059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:50.495234 9059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:50.504097 9065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:50.504091 9067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:50.504257 9064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:50.504338 9059 server_base.cc:1034] running on GCE node
I20260620 04:30:50.505177 9059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:50.506515 9059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:50.507692 9059 hybrid_clock.cc:648] HybridClock initialized: now 1781929850507666 us; error 43 us; skew 500 ppm
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:50.510636 9059 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:30:50.511608 9059 webserver.cc:460] Webserver started at http://127.0.98.62:41089/ using document root <none> and password file <none>
I20260620 04:30:50.512173 9059 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:50.512247 9059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:50.522401 9059 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260620 04:30:50.525328 9074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.526687 9059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:30:50.526850 9059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "7fb660a93d22592dcacbc62ecbf1d78a"
server_key_iv: "555ef903fa7e0f8033725cb5d032d8b1"
server_key_version: "encryptionkey@0"
I20260620 04:30:50.527428 9059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:50.541590 9059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:50.544613 9059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:50.544809 9059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:50.551683 9059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35355
I20260620 04:30:50.551713 9126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35355 every 8 connection(s)
I20260620 04:30:50.552744 9059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:30:50.557149 9127 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:30:50.558811 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7: Bootstrap starting.
I20260620 04:30:50.561367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 9059
I20260620 04:30:50.561839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33719
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35355
--webserver_port=41301
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:30:50.580256 9127 log.cc:824] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:30:50.587702 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:30:50.588066 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7: Bootstrap complete.
I20260620 04:30:50.592120 9127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260620 04:30:50.592554 9127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5930e4ad0cd045b5b2edbf9bbc9c59f7, State: Initialized, Role: FOLLOWER
I20260620 04:30:50.593122 9127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260620 04:30:50.593281 9127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:30:50.593353 9127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:30:50.593452 9127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:30:50.595434 9127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260620 04:30:50.595763 9127 leader_election.cc:302] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5930e4ad0cd045b5b2edbf9bbc9c59f7; no voters:
I20260620 04:30:50.596077 9127 leader_election.cc:288] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:30:50.596242 9131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:30:50.596479 9131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [term 2 LEADER]: Becoming Leader. State: Replica: 5930e4ad0cd045b5b2edbf9bbc9c59f7, State: Running, Role: LEADER
I20260620 04:30:50.596803 9131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } }
I20260620 04:30:50.597263 9127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:30:50.597875 9131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5930e4ad0cd045b5b2edbf9bbc9c59f7. Latest consensus state: current_term: 2 leader_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } } }
I20260620 04:30:50.598011 9131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:50.598369 9132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5930e4ad0cd045b5b2edbf9bbc9c59f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35355 } } }
I20260620 04:30:50.598548 9132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:30:50.598381 9139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:30:50.602259 9139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:30:50.602965 9139 catalog_manager.cc:1259] Loaded cluster ID: eb484e4555d54eca9cab6de82807db1b
I20260620 04:30:50.603027 9139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:30:50.605387 9139 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:30:50.606060 9139 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5930e4ad0cd045b5b2edbf9bbc9c59f7: Loaded TSK: 0
I20260620 04:30:50.607311 9139 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:30:50.682103 9129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:50.682380 9129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:50.682444 9129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:50.686961 9129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:50.687054 9129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:50.687146 9129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:30:50.692765 9129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41301
--tserver_master_addrs=127.0.98.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:50.693897 9129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:50.694936 9129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:50.706204 9153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:50.706209 9154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:50.706828 9129 server_base.cc:1034] running on GCE node
W20260620 04:30:50.707026 9156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:50.707554 9129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:50.708460 9129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:50.709666 9129 hybrid_clock.cc:648] HybridClock initialized: now 1781929850709626 us; error 52 us; skew 500 ppm
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929850, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:50.713407 9129 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 04:30:50.714728 9129 webserver.cc:460] Webserver started at http://127.0.98.1:41301/ using document root <none> and password file <none>
I20260620 04:30:50.715479 9129 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:50.715588 9129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:50.720767 9129 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:30:50.723258 9163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.724790 9129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:50.724923 9129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "a383aab570d243739de4ef93cdbeeeb9"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "26955fb968cbbd156e9b8c687be3ae5e"
server_key_iv: "e5aa5f915f06181a6642e8f2fb8d1838"
server_key_version: "encryptionkey@0"
I20260620 04:30:50.725662 9129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:50.748572 9129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:50.751884 9129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:50.752038 9129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:50.752586 9129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:50.753427 9129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:50.753474 9129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.753512 9129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:50.753526 9129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.761919 9129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33719
I20260620 04:30:50.761996 9276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33719 every 8 connection(s)
I20260620 04:30:50.762900 9129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:30:50.769706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 9129
I20260620 04:30:50.770187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43091
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35355
--webserver_port=43329
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:50.777252 9280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:50.764961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:44499)
0620 04:30:50.765446 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:50.765498 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:50.766760 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:50.767956 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:50.767962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:50.768325 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:50.768794 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.768809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.770572 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.770575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.771547 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.771559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.772356 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.772361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.772538 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.773221 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:50.773239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:50.776019 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":124,"threads_started":1}
W20260620 04:30:50.777316 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:50.764892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44499 (local address 127.0.98.62:35355)
0620 04:30:50.765336 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:50.765351 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:50.765641 (+ 290us) server_negotiation.cc:407] Connection header received
0620 04:30:50.767210 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:50.767222 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:50.767551 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:50.767836 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:50.768984 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.770431 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.771697 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.772185 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.772346 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.776346 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44499: BlockingRecv error: recv got EOF from 127.0.98.1:44499 (error 108)
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":92,"threads_started":1}
W20260620 04:30:50.778043 9277 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35355 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35355: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:50.784299 9280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:50.778506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:35139)
0620 04:30:50.778661 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:50.778677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:50.778773 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:50.779029 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:50.779032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:50.779225 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:50.779357 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.779365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.780079 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.780083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.781084 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.781096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.781625 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.781629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.781724 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.782191 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:50.782205 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:50.784143 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 04:30:50.784543 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:50.778630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35139 (local address 127.0.98.62:35355)
0620 04:30:50.778725 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:50.778728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:50.778742 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:50.778876 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:50.778878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:50.778919 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:50.778983 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:50.779464 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.779972 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.781234 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.781496 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.781599 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.784373 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35139: BlockingRecv error: recv got EOF from 127.0.98.1:35139 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:50.790560 9280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:50.785311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:39565)
0620 04:30:50.785530 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:50.785542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:50.785616 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:50.785839 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:50.785841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:50.785969 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:50.786049 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.786054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.787007 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.787009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.787536 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.787544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.788073 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.788075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.788133 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.788567 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:50.788579 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:50.790451 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155}
W20260620 04:30:50.790726 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:50.785413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39565 (local address 127.0.98.62:35355)
0620 04:30:50.785530 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:50.785534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:50.785582 (+ 48us) server_negotiation.cc:407] Connection header received
0620 04:30:50.785700 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:50.785703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:50.785749 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:50.785820 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:50.786135 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.786913 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.787630 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.787980 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.788065 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.790609 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39565: BlockingRecv error: recv got EOF from 127.0.98.1:39565 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 04:30:50.790869 9277 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:30:50.887003 9281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:50.887295 9281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:50.887344 9281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:50.891441 9281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:50.891521 9281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:50.891584 9281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:30:50.897204 9281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43329
--tserver_master_addrs=127.0.98.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:50.898317 9281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:50.899325 9281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:30:50.909332 9281 server_base.cc:1034] running on GCE node
W20260620 04:30:50.909353 9286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:50.909533 9289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:50.909359 9287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:50.909968 9281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:50.910687 9281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:50.911864 9281 hybrid_clock.cc:648] HybridClock initialized: now 1781929850911840 us; error 43 us; skew 500 ppm
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929850, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:50.914784 9281 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 04:30:50.915732 9281 webserver.cc:460] Webserver started at http://127.0.98.2:43329/ using document root <none> and password file <none>
I20260620 04:30:50.916399 9281 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:50.916484 9281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:50.920588 9281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 04:30:50.922982 9296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.924320 9281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:30:50.924435 9281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "58183efd1d114742a84b82cbbc65c8dc"
format_stamp: "Formatted at 2026-06-20 04:30:49 on dist-test-slave-rzqm"
server_key: "044c09dd7062ab1cdb012bfe4f478d66"
server_key_iv: "a57db82894d74d28ab1cd1992776b7ba"
server_key_version: "encryptionkey@0"
I20260620 04:30:50.924993 9281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:50.938014 9281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:50.941382 9281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:50.941591 9281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:50.942219 9281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:50.943137 9281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:50.943187 9281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.943243 9281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:50.943271 9281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:50.953293 9281 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43091
I20260620 04:30:50.953374 9409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43091 every 8 connection(s)
I20260620 04:30:50.954869 9281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:30:50.956904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 9281
I20260620 04:30:50.957454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39555
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35355
--webserver_port=41629
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:30:50.967844 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:50.957089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53167 (local address 127.0.98.62:35355)
0620 04:30:50.957252 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:50.957257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:50.958255 (+ 998us) server_negotiation.cc:407] Connection header received
0620 04:30:50.959876 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:50.959880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:50.959945 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:50.960026 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:50.961482 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.962016 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.963241 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.963478 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.963577 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.967708 (+ 4131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53167: BlockingRecv error: recv got EOF from 127.0.98.2:53167 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:30:50.968585 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:50.957487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:53167)
0620 04:30:50.958062 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:50.958125 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:50.959618 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:50.960255 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:50.960263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:50.960663 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:50.961316 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.961333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.962154 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.962158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.963110 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.963120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.963653 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.963657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.963857 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.964571 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:50.964589 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:50.967523 (+ 2934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":149,"threads_started":1}
W20260620 04:30:50.969192 9410 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35355 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35355: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:50.976058 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:50.969671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:42251)
0620 04:30:50.969860 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:50.969875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:50.969963 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:50.970240 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:50.970243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:50.970427 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:50.970549 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.970555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.971375 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.971379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.972342 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.972355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.972829 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.972832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.972933 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.973466 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:50.973481 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:50.975909 (+ 2428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119}
W20260620 04:30:50.976425 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:50.969760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42251 (local address 127.0.98.62:35355)
0620 04:30:50.969867 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:50.969870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:50.969916 (+ 46us) server_negotiation.cc:407] Connection header received
0620 04:30:50.970050 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:50.970053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:50.970140 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:50.970212 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:50.970657 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.971245 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.972487 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.972711 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.972802 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.976077 (+ 3275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42251: BlockingRecv error: recv got EOF from 127.0.98.2:42251 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:50 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:50.982172 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:50.976602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:39847)
0620 04:30:50.976771 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:50.976785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:50.976859 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:50.977228 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:50.977231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:50.977398 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:50.977499 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.977506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.978256 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.978259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.979018 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:50.979028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.979535 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.979539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:50.979616 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.980081 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:50.980095 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:50.982029 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20260620 04:30:50.982470 9410 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:30:50.982713 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:50.976697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39847 (local address 127.0.98.62:35355)
0620 04:30:50.976796 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:50.976799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:50.976823 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:30:50.976983 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:50.976986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:50.977075 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:50.977198 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:50.977639 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.978148 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.979184 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:50.979411 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:50.979502 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:50.982347 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39847: BlockingRecv error: recv got EOF from 127.0.98.2:39847 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 04:30:51.076527 9413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:30:51.076840 9413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:30:51.076901 9413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:30:51.081229 9413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:30:51.081336 9413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:30:51.081434 9413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:30:51.087280 9413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41629
--tserver_master_addrs=127.0.98.62:35355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:30:51.088388 9413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:30:51.089468 9413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:30:51.099346 9418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:30:51.099390 9419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:51.099473 9413 server_base.cc:1034] running on GCE node
W20260620 04:30:51.099687 9421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:30:51.100147 9413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:30:51.100867 9413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:30:51.102056 9413 hybrid_clock.cc:648] HybridClock initialized: now 1781929851102020 us; error 51 us; skew 500 ppm
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929851, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:30:51.105041 9413 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 04:30:51.106025 9413 webserver.cc:460] Webserver started at http://127.0.98.3:41629/ using document root <none> and password file <none>
I20260620 04:30:51.106585 9413 fs_manager.cc:359] Metadata directory not provided
I20260620 04:30:51.106673 9413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:30:51.110927 9413 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:30:51.113179 9428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:51.114464 9413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:30:51.114584 9413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "70c40e539ad4421e8d1e7c52b5a6c990"
format_stamp: "Formatted at 2026-06-20 04:30:50 on dist-test-slave-rzqm"
server_key: "49ecd4aeaf581929a315871ad9360bcb"
server_key_iv: "b203d8116da20846cf58825f0805926e"
server_key_version: "encryptionkey@0"
I20260620 04:30:51.115156 9413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:30:51.147763 9413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:30:51.151494 9413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:30:51.151696 9413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:30:51.152330 9413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:30:51.153251 9413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:30:51.153304 9413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:51.153359 9413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:30:51.153388 9413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:30:51.162565 9413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39555
I20260620 04:30:51.162636 9541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39555 every 8 connection(s)
I20260620 04:30:51.163718 9413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:30:51.164691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 9413
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:51.175196 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:51.165527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.0.98.62:35355)
0620 04:30:51.166095 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:51.166099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:51.166114 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:51.166162 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:51.166165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:51.166213 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:51.166308 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:51.167202 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.167743 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.168612 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.168874 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.168967 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.170672 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:30:51.170681 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:30:51.170702 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:30:51.170740 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:30:51.172628 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:51.173136 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:51.173143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:51.173151 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:51.173314 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:30:51.173645 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:30:51.173649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:30:51.173651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:30:51.174122 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:30:51.174320 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 04:30:51.174692 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 04:30:51.174944 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464}
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 04:30:51.177731 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:51.166353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36555 (local address 127.0.98.62:35355)
0620 04:30:51.166584 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:51.166588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:51.167375 (+ 787us) server_negotiation.cc:407] Connection header received
0620 04:30:51.168786 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:51.168789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:51.168843 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:51.168942 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:51.170719 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.171451 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.172767 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.173040 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.173139 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.177545 (+ 4406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36555: BlockingRecv error: recv got EOF from 127.0.98.3:36555 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"spinlock_wait_cycles":768,"thread_start_us":85,"threads_started":1}
I20260620 04:30:51.178720 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:51.166695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:36555)
0620 04:30:51.167215 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:51.167272 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:51.168589 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:51.169281 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:51.169292 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:51.169826 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:51.170576 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:51.170598 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.171611 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.171615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:51.172630 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:51.172641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.173192 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.173195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:51.173367 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.174050 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:51.174067 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:51.177381 (+ 3314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":132,"threads_started":1}
W20260620 04:30:51.179198 9542 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35355 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35355: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:51.185586 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:51.179718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:37445)
0620 04:30:51.179853 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:51.179865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:51.179969 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:51.180208 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:51.180210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:51.180373 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:51.180472 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:51.180478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.181397 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.181401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:51.182028 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:51.182041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.182599 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.182602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:51.182686 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.183168 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:51.183180 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:51.185458 (+ 2278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 04:30:51.185784 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:51.179843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37445 (local address 127.0.98.62:35355)
0620 04:30:51.179976 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:51.179980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:51.179994 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:51.180061 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:51.180063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:51.180107 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:51.180176 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:51.180597 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.181266 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.182150 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.182388 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.182483 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.185651 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37445: BlockingRecv error: recv got EOF from 127.0.98.3:37445 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:51.191471 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:51.186196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:51621)
0620 04:30:51.186317 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:51.186331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:51.186403 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:51.186753 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:51.186756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:51.186909 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:51.186991 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:51.186996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.187735 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.187738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:51.188316 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:51.188325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.188771 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.188774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:51.188852 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.189364 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:51.189377 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:51.191327 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 04:30:51.191563 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:51.186362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51621 (local address 127.0.98.62:35355)
0620 04:30:51.186536 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:51.186540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:51.186562 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:30:51.186608 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:51.186610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:51.186650 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:51.186716 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:51.187101 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.187631 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.188434 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.188664 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.188753 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.191464 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51621: BlockingRecv error: recv got EOF from 127.0.98.3:51621 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260620 04:30:51.191708 9542 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:51.797520 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:51.791611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:59319)
0620 04:30:51.791913 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:51.791930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:51.792031 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:51.792306 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:51.792309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:51.792502 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:51.792622 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:51.792628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.793462 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.793465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:51.794195 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:51.794206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.794731 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.794734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:51.794824 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.795328 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:51.795340 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:51.797380 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":122,"threads_started":1}
W20260620 04:30:51.797646 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:51.791746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59319 (local address 127.0.98.62:35355)
0620 04:30:51.791969 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:51.791972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:51.791987 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:51.792114 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:51.792116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:51.792159 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:51.792239 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:51.792743 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.793330 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.794310 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.794616 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.794712 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.797529 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59319: BlockingRecv error: recv got EOF from 127.0.98.1:59319 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:51 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:51.989442 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:51.983347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:59497)
0620 04:30:51.983645 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:51.983660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:51.983791 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:51.984070 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:51.984073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:51.984306 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:51.984449 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:51.984457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.985368 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.985371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:51.985959 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:51.985969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.986434 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.986437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:51.986520 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.987129 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:51.987141 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:51.989307 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
W20260620 04:30:51.989583 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:51.983475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59497 (local address 127.0.98.62:35355)
0620 04:30:51.983669 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:51.983673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:51.983737 (+ 64us) server_negotiation.cc:407] Connection header received
0620 04:30:51.983871 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:51.983873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:51.983919 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:51.983992 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:51.984603 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.985228 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.986070 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:51.986320 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:51.986405 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:51.989473 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59497: BlockingRecv error: recv got EOF from 127.0.98.2:59497 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 04:30:52 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:52 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:52.198307 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:52.192503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:46535)
0620 04:30:52.192843 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:52.192858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:52.192976 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:52.193367 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:52.193370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:52.193560 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:52.193683 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:52.193689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.194462 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.194464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:52.195055 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:52.195067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.195590 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.195594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:52.195681 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:52.196216 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:52.196232 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:52.198177 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
W20260620 04:30:52.198470 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:52.192702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46535 (local address 127.0.98.62:35355)
0620 04:30:52.192858 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:52.192861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:52.192932 (+ 71us) server_negotiation.cc:407] Connection header received
0620 04:30:52.193110 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:52.193113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:52.193212 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:52.193281 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:52.193800 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.194353 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.195199 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.195481 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.195560 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:52.198328 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46535: BlockingRecv error: recv got EOF from 127.0.98.3:46535 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 04:30:52 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:52 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:52.804586 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:52.798464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:38735)
0620 04:30:52.798763 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:52.798777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:52.798889 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:52.799284 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:52.799288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:52.799515 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:52.799632 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:52.799639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.800428 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.800431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:52.801164 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:52.801177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.801844 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.801848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:52.801946 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:52.802473 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:52.802486 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:52.804444 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
W20260620 04:30:52.804716 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:52.798550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38735 (local address 127.0.98.62:35355)
0620 04:30:52.798826 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:52.798830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:52.798844 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:52.799014 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:52.799017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:52.799078 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:52.799195 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:52.799740 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.800264 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.801279 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.801595 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.801675 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:52.804598 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38735: BlockingRecv error: recv got EOF from 127.0.98.1:38735 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
Jun 20 04:30:52 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:52 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:52.995889 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:52.990296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:50395)
0620 04:30:52.990584 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:52.990600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:52.990735 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:52.990979 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:52.990982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:52.991194 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:52.991289 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:52.991295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.992081 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.992085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:52.992655 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:52.992662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.993165 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.993168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:52.993249 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:52.993764 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:52.993776 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:52.995748 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
W20260620 04:30:52.996121 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:52.990409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50395 (local address 127.0.98.62:35355)
0620 04:30:52.990569 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:52.990574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:52.990685 (+ 111us) server_negotiation.cc:407] Connection header received
0620 04:30:52.990824 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:52.990828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:52.990874 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:52.990942 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:52.991405 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.991957 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.992764 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:52.992983 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:52.993111 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:52.995929 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50395: BlockingRecv error: recv got EOF from 127.0.98.2:50395 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 04:30:53 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:53 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:53.204886 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:53.199221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:51137)
0620 04:30:53.199511 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:53.199528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:53.199660 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:53.199936 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:53.199939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:53.200130 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:53.200235 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:53.200241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.200970 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:53.200973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:53.201556 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:53.201566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.201991 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:53.201995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:53.202075 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:53.202582 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:53.202594 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:53.204705 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
W20260620 04:30:53.205060 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:53.199417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51137 (local address 127.0.98.62:35355)
0620 04:30:53.199553 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:53.199557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:53.199599 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:30:53.199787 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:53.199790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:53.199834 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:53.199899 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:53.200356 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.200867 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:53.201675 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.201890 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:53.201977 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:53.204876 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51137: BlockingRecv error: recv got EOF from 127.0.98.3:51137 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 04:30:53 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:53 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:53.812011 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:53.805538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:38339)
0620 04:30:53.805826 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:53.805841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:53.805959 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:53.806328 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:53.806331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:53.806609 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:53.806714 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:53.806721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.807855 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:53.807858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:53.808456 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:53.808466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.808945 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:53.808948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:53.809083 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:53.809678 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:53.809691 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:53.811868 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
W20260620 04:30:53.812198 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:53.805710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38339 (local address 127.0.98.62:35355)
0620 04:30:53.805997 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:53.806001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:53.806016 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:53.806058 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:53.806060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:53.806108 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:53.806240 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:53.806874 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.807672 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:53.808582 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.808833 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:53.808922 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:53.812043 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38339: BlockingRecv error: recv got EOF from 127.0.98.1:38339 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":105,"threads_started":1}
Jun 20 04:30:54 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:54 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:54.002846 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:53.996871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:59871)
0620 04:30:53.997148 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:53.997164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:53.997288 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:53.997611 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:53.997613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:53.997804 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:53.997907 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:53.997912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.998923 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:53.998926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:53.999587 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:53.999598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:54.000059 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:54.000063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:54.000144 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:54.000734 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:54.000746 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:54.002651 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":144,"threads_started":1}
W20260620 04:30:54.002921 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:53.997026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59871 (local address 127.0.98.62:35355)
0620 04:30:53.997146 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:53.997150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:53.997235 (+ 85us) server_negotiation.cc:407] Connection header received
0620 04:30:53.997395 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:53.997398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:53.997448 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:53.997514 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:53.998111 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.998790 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:53.999715 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:53.999950 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:54.000043 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:54.002818 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59871: BlockingRecv error: recv got EOF from 127.0.98.2:59871 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 04:30:54 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:54 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:54.211392 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:54.205834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:54907)
0620 04:30:54.206140 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:54.206180 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:54.206299 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:54.206561 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:54.206564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:54.206804 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:54.206904 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:54.206910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:54.207701 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:54.207705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:54.208298 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:54.208308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:54.208734 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:54.208737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:54.208831 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:54.209380 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:54.209393 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:54.211205 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
W20260620 04:30:54.211498 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:54.205957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54907 (local address 127.0.98.62:35355)
0620 04:30:54.206131 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:54.206136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:54.206253 (+ 117us) server_negotiation.cc:407] Connection header received
0620 04:30:54.206400 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:54.206403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:54.206453 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:54.206521 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:54.207026 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:54.207579 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:54.208412 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:54.208629 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:54.208716 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:54.211363 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54907: BlockingRecv error: recv got EOF from 127.0.98.3:54907 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 04:30:54 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:54 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:54.821915 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:54.812768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:38521)
0620 04:30:54.813098 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:54.813116 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:54.813237 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:54.813518 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:54.813520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:54.813702 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:54.813812 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:54.813819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:54.814683 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:54.814685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:54.815271 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:54.815280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:54.815739 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:54.815741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:54.815806 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:54.819792 (+ 3986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:54.819809 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:54.821777 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
W20260620 04:30:54.822088 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:54.812920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38521 (local address 127.0.98.62:35355)
0620 04:30:54.813289 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:54.813293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:54.813307 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:54.813352 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:54.813354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:54.813397 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:54.813479 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:54.813949 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:54.814547 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:54.815396 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:54.815626 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:54.815720 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:54.821947 (+ 6227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38521: BlockingRecv error: recv got EOF from 127.0.98.1:38521 (error 108)
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
Jun 20 04:30:55 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:55 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:55.009022 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:55.003727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:48195)
0620 04:30:55.004026 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:55.004040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:55.004161 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:55.004402 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:55.004404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:55.004597 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:55.004699 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:55.004705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.005509 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.005513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:55.006089 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:55.006099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.006517 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.006520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:55.006607 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:55.007102 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:55.007114 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:55.008868 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":88,"threads_started":1}
W20260620 04:30:55.009137 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:55.003833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48195 (local address 127.0.98.62:35355)
0620 04:30:55.004008 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:55.004013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:55.004173 (+ 160us) server_negotiation.cc:407] Connection header received
0620 04:30:55.004250 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:55.004252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:55.004297 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:55.004366 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:55.004816 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.005394 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.006207 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.006416 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.006503 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:55.009035 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48195: BlockingRecv error: recv got EOF from 127.0.98.2:48195 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 04:30:55 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:55 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:55.217674 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:55.212258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:54967)
0620 04:30:55.212571 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:55.212587 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:55.212713 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:55.212966 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:55.212969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:55.213231 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:55.213330 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:55.213336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.214096 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.214100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:55.214651 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:55.214660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.215106 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.215110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:55.215190 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:55.215699 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:55.215711 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:55.217507 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":87,"threads_started":1}
W20260620 04:30:55.217772 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:55.212360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54967 (local address 127.0.98.62:35355)
0620 04:30:55.212572 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:55.212577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:55.212667 (+ 90us) server_negotiation.cc:407] Connection header received
0620 04:30:55.212808 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:55.212811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:55.212859 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:55.212927 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:55.213458 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.213986 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.214778 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.214999 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.215089 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:55.217668 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54967: BlockingRecv error: recv got EOF from 127.0.98.3:54967 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 04:30:55 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:55 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:55.828476 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:55.822787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:37095)
0620 04:30:55.823057 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:55.823070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:55.823162 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:55.823432 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:55.823434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:55.823621 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:55.823715 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:55.823721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.824594 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.824597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:55.825220 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:55.825230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.825785 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.825788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:55.825852 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:55.826332 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:55.826343 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:55.828301 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
W20260620 04:30:55.828523 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:55.822854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37095 (local address 127.0.98.62:35355)
0620 04:30:55.823120 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:55.823124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:55.823138 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:55.823260 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:55.823262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:55.823304 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:55.823384 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:55.823863 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.824469 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.825365 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:55.825658 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:55.825757 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:55.828419 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37095: BlockingRecv error: recv got EOF from 127.0.98.1:37095 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
Jun 20 04:30:56 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:56 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:56.015134 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:56.009851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:35875)
0620 04:30:56.010140 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:56.010154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:56.010255 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:56.010497 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:56.010499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:56.010681 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:56.010779 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:56.010784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.011519 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.011522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:56.012100 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:56.012110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.012580 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.012582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:56.012641 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:56.013220 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:56.013232 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:56.014969 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
W20260620 04:30:56.015215 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:56.010034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35875 (local address 127.0.98.62:35355)
0620 04:30:56.010160 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:56.010164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:56.010235 (+ 71us) server_negotiation.cc:407] Connection header received
0620 04:30:56.010327 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:56.010330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:56.010372 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:56.010444 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:56.010872 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.011395 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.012239 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.012482 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.012573 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:56.015115 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35875: BlockingRecv error: recv got EOF from 127.0.98.2:35875 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 04:30:56 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:56 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:56.228113 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:56.218696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:51609)
0620 04:30:56.218950 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:56.218966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:56.219077 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:56.219351 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:56.219354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:56.219567 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:56.219670 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:56.219675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.220468 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.220472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:56.221247 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:56.221256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.221681 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.221686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:56.221778 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:56.225820 (+ 4042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:56.225835 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:56.227951 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
W20260620 04:30:56.228277 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:56.218793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51609 (local address 127.0.98.62:35355)
0620 04:30:56.218952 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:56.218957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:56.219035 (+ 78us) server_negotiation.cc:407] Connection header received
0620 04:30:56.219177 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:56.219180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:56.219248 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:56.219314 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:56.219817 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.220363 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.221347 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.221571 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.221675 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:56.228151 (+ 6476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51609: BlockingRecv error: recv got EOF from 127.0.98.3:51609 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 04:30:56 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:56 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:56.835304 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:56.829386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:59761)
0620 04:30:56.829702 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:56.829716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:56.829826 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:56.830195 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:56.830199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:56.830388 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:56.830515 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:56.830523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.831520 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.831523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:56.832208 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:56.832219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.832639 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.832641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:56.832699 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:56.833280 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:56.833293 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:56.835178 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":116,"threads_started":1}
W20260620 04:30:56.835508 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:56.829525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59761 (local address 127.0.98.62:35355)
0620 04:30:56.829791 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:56.829795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:56.829810 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:56.829966 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:56.829969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:56.830027 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:56.830120 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:56.830656 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.831414 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.832309 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:56.832550 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:56.832632 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:56.835353 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59761: BlockingRecv error: recv got EOF from 127.0.98.1:59761 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
Jun 20 04:30:57 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:57 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:57.021713 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:57.016029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:37423)
0620 04:30:57.016359 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:57.016373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:57.016475 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:57.016725 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:57.016728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:57.016917 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:57.017050 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:57.017058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.017805 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.017808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:57.018404 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:57.018412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.018858 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.018860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:57.018918 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:57.019436 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:57.019447 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:57.021574 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":124,"threads_started":1}
W20260620 04:30:57.021807 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:57.016140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37423 (local address 127.0.98.62:35355)
0620 04:30:57.016295 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:57.016300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:57.016466 (+ 166us) server_negotiation.cc:407] Connection header received
0620 04:30:57.016544 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:57.016547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:57.016597 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:57.016695 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:57.017162 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.017713 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.018501 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.018726 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.018839 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:57.021691 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37423: BlockingRecv error: recv got EOF from 127.0.98.2:37423 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 04:30:57 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:57 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:57.234558 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:57.228951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:50611)
0620 04:30:57.229241 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:57.229255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:57.229352 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:57.229606 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:57.229608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:57.229777 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:57.229872 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:57.229877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.230754 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.230757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:57.231324 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:57.231333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.231851 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.231853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:57.231932 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:57.232419 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:57.232431 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:57.234425 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":120,"threads_started":1}
W20260620 04:30:57.234762 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:57.229079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50611 (local address 127.0.98.62:35355)
0620 04:30:57.229200 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:57.229204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:57.229312 (+ 108us) server_negotiation.cc:407] Connection header received
0620 04:30:57.229446 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:57.229448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:57.229491 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:57.229577 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:57.229993 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.230594 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.231461 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.231736 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.231838 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:57.234640 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50611: BlockingRecv error: recv got EOF from 127.0.98.3:50611 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 04:30:57 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:57 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:57.841192 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:57.836046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:35717)
0620 04:30:57.836381 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:57.836394 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:57.836496 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:57.836729 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:57.836731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:57.836909 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:57.837032 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:57.837038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.837762 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.837765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:57.838321 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:57.838328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.838734 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.838738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:57.838807 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:57.839275 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:57.839286 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:57.841038 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":85,"threads_started":1}
W20260620 04:30:57.841305 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:57.836144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35717 (local address 127.0.98.62:35355)
0620 04:30:57.836373 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:57.836377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:57.836466 (+ 89us) server_negotiation.cc:407] Connection header received
0620 04:30:57.836570 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:57.836572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:57.836613 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:57.836686 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:57.837135 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.837665 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.838413 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:57.838640 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:57.838718 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:57.841181 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35717: BlockingRecv error: recv got EOF from 127.0.98.1:35717 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
Jun 20 04:30:58 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:58 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:58.028247 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:58.022571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:47413)
0620 04:30:58.022911 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:58.022924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:58.023031 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:58.023487 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:58.023490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:58.023692 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:58.023789 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:58.023795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.024584 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.024587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:58.025241 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:58.025251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.025740 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.025742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:58.025800 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:58.026298 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:58.026309 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:58.028131 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":80,"threads_started":1}
W20260620 04:30:58.028425 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:58.022663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47413 (local address 127.0.98.62:35355)
0620 04:30:58.022807 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:58.022812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:58.022992 (+ 180us) server_negotiation.cc:407] Connection header received
0620 04:30:58.023298 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:58.023301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:58.023367 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:58.023440 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:58.023933 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.024478 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.025362 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.025581 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.025728 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:58.028277 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47413: BlockingRecv error: recv got EOF from 127.0.98.2:47413 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 04:30:58 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:58 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:58.240888 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:58.235416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:47181)
0620 04:30:58.235839 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:58.235860 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:58.235998 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:58.236285 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:58.236288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:58.236506 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:58.236661 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:58.236668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.237418 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.237422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:58.238027 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:58.238037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.238426 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.238428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:58.238493 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:58.239033 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:58.239048 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:58.240771 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":81,"threads_started":1}
W20260620 04:30:58.241065 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:58.235484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47181 (local address 127.0.98.62:35355)
0620 04:30:58.235734 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:58.235738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:58.235957 (+ 219us) server_negotiation.cc:407] Connection header received
0620 04:30:58.236098 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:58.236101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:58.236160 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:58.236229 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:58.236764 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.237323 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.238125 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.238330 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.238401 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:58.240918 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47181: BlockingRecv error: recv got EOF from 127.0.98.3:47181 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Jun 20 04:30:58 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:58 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:58.847636 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:58.842085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:38899)
0620 04:30:58.842364 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:58.842381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:58.842483 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:58.842730 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:58.842732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:58.842952 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:58.843060 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:58.843066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.843907 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.843910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:58.844487 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:58.844496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.844992 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.844995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:58.845132 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:58.845660 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:58.845672 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:58.847510 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
W20260620 04:30:58.847841 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:58.842219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38899 (local address 127.0.98.62:35355)
0620 04:30:58.842459 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:58.842463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:58.842478 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:30:58.842565 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:58.842567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:58.842608 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:58.842699 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:58.843198 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.843778 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.844616 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:58.844860 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:58.844948 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:58.847685 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38899: BlockingRecv error: recv got EOF from 127.0.98.1:38899 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
Jun 20 04:30:59 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:59 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:59.034844 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:59.029149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:55775)
0620 04:30:59.029448 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:59.029462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:59.029570 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:59.029892 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:59.029895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:59.030074 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:59.030230 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:59.030238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.031224 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.031227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:59.031882 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:59.031894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.032361 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.032365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:59.032429 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:59.032926 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:59.032937 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:59.034724 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":122,"threads_started":1}
W20260620 04:30:59.034968 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:59.029204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55775 (local address 127.0.98.62:35355)
0620 04:30:59.029336 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:59.029340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:59.029556 (+ 216us) server_negotiation.cc:407] Connection header received
0620 04:30:59.029642 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:59.029645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:59.029695 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:59.029774 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:59.030364 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.031112 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.032023 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.032269 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.032390 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:59.034863 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55775: BlockingRecv error: recv got EOF from 127.0.98.2:55775 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 04:30:59 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:59 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:59.247434 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:59.241826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:40357)
0620 04:30:59.242120 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:59.242133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:59.242233 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:59.242512 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:59.242515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:59.242694 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:59.242803 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:59.242809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.243618 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.243621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:59.244347 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:59.244359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.244861 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.244864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:59.244940 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:59.245567 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:59.245580 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:59.247287 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
W20260620 04:30:59.247534 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:59.241994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40357 (local address 127.0.98.62:35355)
0620 04:30:59.242184 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:59.242188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:59.242205 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:30:59.242307 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:59.242309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:59.242356 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:59.242439 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:59.242952 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.243476 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.244468 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.244750 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.244844 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:59.247432 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40357: BlockingRecv error: recv got EOF from 127.0.98.3:40357 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 04:30:59 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:30:59 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:30:59.856846 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:30:59.848548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:54717)
0620 04:30:59.848945 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 04:30:59.848964 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:30:59.849168 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:30:59.849471 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:30:59.849474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:30:59.849651 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:30:59.849746 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:59.849752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.850530 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.850533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:59.851112 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:30:59.851121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.851598 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.851601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:30:59.851668 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:59.854659 (+ 2991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:30:59.854673 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:30:59.856709 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":91,"threads_started":1}
W20260620 04:30:59.857048 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:30:59.848800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54717 (local address 127.0.98.62:35355)
0620 04:30:59.849083 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 04:30:59.849087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:30:59.849101 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:30:59.849276 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:30:59.849279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:30:59.849322 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:30:59.849396 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:30:59.849873 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.850411 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.851235 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:30:59.851490 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:30:59.851575 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:30:59.856883 (+ 5308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54717: BlockingRecv error: recv got EOF from 127.0.98.1:54717 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
Jun 20 04:31:00 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:00 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:00.040959 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:00.035526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:44237)
0620 04:31:00.035802 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:00.035818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:00.035918 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:00.036126 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:00.036128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:00.036308 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:00.036403 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:00.036408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.037315 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.037318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:00.037913 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:00.037923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.038411 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.038414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:00.038486 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:00.039018 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:00.039030 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:00.040838 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
W20260620 04:31:00.041128 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:00.035600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44237 (local address 127.0.98.62:35355)
0620 04:31:00.035791 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:00.035795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:00.035881 (+ 86us) server_negotiation.cc:407] Connection header received
0620 04:31:00.035986 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:00.035988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:00.036030 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:00.036099 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:00.036536 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.037131 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.038037 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.038265 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.038347 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:00.040985 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44237: BlockingRecv error: recv got EOF from 127.0.98.2:44237 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 04:31:00 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:00 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:00.253968 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:00.248330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:44751)
0620 04:31:00.248648 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:00.248662 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:00.248765 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:00.249066 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:00.249069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:00.249295 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:00.249394 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:00.249400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.250212 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.250215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:00.250781 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:00.250789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.251211 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.251215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:00.251292 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:00.251815 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:00.251827 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:00.253815 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
W20260620 04:31:00.254098 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:00.248458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44751 (local address 127.0.98.62:35355)
0620 04:31:00.248620 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:00.248625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:00.248734 (+ 109us) server_negotiation.cc:407] Connection header received
0620 04:31:00.248863 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:00.248866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:00.248917 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:00.249023 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:00.249507 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.250105 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.250889 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.251108 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.251198 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:00.253963 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44751: BlockingRecv error: recv got EOF from 127.0.98.3:44751 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 04:31:00 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:00 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:00.863111 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:00.857818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:37835)
0620 04:31:00.858088 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:00.858102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:00.858195 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:00.858496 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:00.858499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:00.858675 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:00.858777 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:00.858782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.859607 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.859611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:00.860186 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:00.860195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.860644 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.860646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:00.860711 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:00.861247 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:00.861257 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:00.862993 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
W20260620 04:31:00.863312 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:00.857946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37835 (local address 127.0.98.62:35355)
0620 04:31:00.858195 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:00.858199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:00.858214 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:31:00.858256 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:00.858259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:00.858298 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:00.858398 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:00.858929 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.859488 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.860312 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:00.860535 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:00.860617 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:00.863151 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37835: BlockingRecv error: recv got EOF from 127.0.98.1:37835 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
Jun 20 04:31:01 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:01 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:01.046993 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:01.041866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:50001)
0620 04:31:01.042117 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:01.042132 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:01.042235 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:01.042481 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:01.042484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:01.042678 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:01.042769 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:01.042774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.043511 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.043515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:01.044075 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:01.044083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.044515 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.044518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:01.044594 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:01.045070 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:01.045081 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:01.046844 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
W20260620 04:31:01.047094 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:01.041976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50001 (local address 127.0.98.62:35355)
0620 04:31:01.042116 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:01.042121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:01.042209 (+ 88us) server_negotiation.cc:407] Connection header received
0620 04:31:01.042332 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:01.042335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:01.042383 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:01.042444 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:01.042881 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.043405 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.044185 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.044417 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.044503 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:01.046994 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50001: BlockingRecv error: recv got EOF from 127.0.98.2:50001 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 04:31:01 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:01 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:01.260262 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:01.254878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:37845)
0620 04:31:01.255192 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:01.255206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:01.255315 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:01.255534 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:01.255537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:01.255711 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:01.255825 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:01.255831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.256614 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.256617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:01.257214 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:01.257232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.257661 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.257664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:01.257734 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:01.258328 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:01.258341 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:01.260140 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":94,"threads_started":1}
W20260620 04:31:01.260530 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:01.255094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37845 (local address 127.0.98.62:35355)
0620 04:31:01.255210 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:01.255214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:01.255259 (+ 45us) server_negotiation.cc:407] Connection header received
0620 04:31:01.255383 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:01.255385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:01.255430 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:01.255495 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:01.255944 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.256491 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.257323 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.257535 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.257614 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:01.260297 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37845: BlockingRecv error: recv got EOF from 127.0.98.3:37845 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 04:31:01 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:01 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:01.869513 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:01.864012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:52485)
0620 04:31:01.864286 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:01.864299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:01.864385 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:01.864624 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:01.864626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:01.864821 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:01.864924 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:01.864930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.865734 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.865737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:01.866336 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:01.866345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.866792 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.866794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:01.866855 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:01.867365 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:01.867377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:01.869382 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
W20260620 04:31:01.869674 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:01.864121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52485 (local address 127.0.98.62:35355)
0620 04:31:01.864350 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:01.864354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:01.864366 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:31:01.864466 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:01.864469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:01.864511 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:01.864591 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:01.865070 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.865616 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.866464 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:01.866692 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:01.866792 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:01.869534 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52485: BlockingRecv error: recv got EOF from 127.0.98.1:52485 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
Jun 20 04:31:02 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:02 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:02.053174 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:02.047823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:36351)
0620 04:31:02.048094 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:02.048108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:02.048210 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:02.048433 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:02.048435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:02.048616 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:02.048712 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:02.048717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.049539 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.049542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:02.050125 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:02.050133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.050606 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.050609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:02.050679 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:02.051203 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:02.051215 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:02.052981 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
W20260620 04:31:02.053267 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:02.047941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36351 (local address 127.0.98.62:35355)
0620 04:31:02.048100 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:02.048104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:02.048173 (+ 69us) server_negotiation.cc:407] Connection header received
0620 04:31:02.048282 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:02.048284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:02.048331 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:02.048401 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:02.048853 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.049426 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.050255 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.050477 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.050568 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:02.053156 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36351: BlockingRecv error: recv got EOF from 127.0.98.2:36351 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 04:31:02 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:02 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:02.266367 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:02.260976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:60289)
0620 04:31:02.261271 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:02.261287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:02.261391 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:02.261606 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:02.261608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:02.261810 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:02.261908 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:02.261914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.262608 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.262610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:02.263192 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:02.263200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.263684 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.263687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:02.263759 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:02.264230 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:02.264242 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:02.266240 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":78,"threads_started":1}
W20260620 04:31:02.266624 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:02.261113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60289 (local address 127.0.98.62:35355)
0620 04:31:02.261258 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:02.261261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:02.261349 (+ 88us) server_negotiation.cc:407] Connection header received
0620 04:31:02.261462 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:02.261464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:02.261508 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:02.261575 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:02.261997 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.262523 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.263280 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.263484 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.263554 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:02.266422 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60289: BlockingRecv error: recv got EOF from 127.0.98.3:60289 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 04:31:02 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:02 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:02.875476 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:02.870306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:47643)
0620 04:31:02.870585 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:02.870600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:02.870696 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:02.870927 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:02.870929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:02.871126 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:02.871223 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:02.871229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.871951 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.871954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:02.872531 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:02.872538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.872924 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.872926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:02.873027 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:02.873517 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:02.873527 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:02.875325 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
W20260620 04:31:02.875756 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:02.870419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47643 (local address 127.0.98.62:35355)
0620 04:31:02.870671 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:02.870674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:02.870690 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:31:02.870776 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:02.870779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:02.870821 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:02.870905 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:02.871322 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.871851 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.872622 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:02.872839 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:02.872931 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:02.875520 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47643: BlockingRecv error: recv got EOF from 127.0.98.1:47643 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
Jun 20 04:31:03 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:03 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:03.059754 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:03.054055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:36599)
0620 04:31:03.054326 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:03.054341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:03.054451 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:03.054791 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:03.054794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:03.054978 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:03.055077 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:03.055083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.055820 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.055823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:03.056513 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:03.056523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.057082 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.057086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:03.057169 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:03.057737 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:03.057749 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:03.059633 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
W20260620 04:31:03.059897 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:03.054153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36599 (local address 127.0.98.62:35355)
0620 04:31:03.054314 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:03.054318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:03.054411 (+ 93us) server_negotiation.cc:407] Connection header received
0620 04:31:03.054538 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:03.054540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:03.054599 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:03.054718 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:03.055182 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.055708 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.056608 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.056906 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.057050 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:03.059782 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36599: BlockingRecv error: recv got EOF from 127.0.98.2:36599 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 04:31:03 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:03 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:03.273132 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:03.267201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:46443)
0620 04:31:03.267475 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:03.267489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:03.267588 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:03.267820 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:03.267822 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:03.268009 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:03.268187 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:03.268196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.268959 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.268962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:03.269602 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:03.269612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.270065 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.270069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:03.270176 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:03.270763 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:03.270775 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:03.272958 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
W20260620 04:31:03.273278 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:03.267384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46443 (local address 127.0.98.62:35355)
0620 04:31:03.267526 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:03.267530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:03.267545 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:31:03.267671 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:03.267673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:03.267720 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:03.267790 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:03.268328 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.268854 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.269715 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.269936 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.270020 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:03.273127 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46443: BlockingRecv error: recv got EOF from 127.0.98.3:46443 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 04:31:03 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:03 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:03.881712 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:03.876203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:44997)
0620 04:31:03.876484 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:03.876500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:03.876595 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:03.876804 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:03.876807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:03.876985 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:03.877111 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:03.877117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.877850 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.877854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:03.878438 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:03.878447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.878967 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.878970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:03.879040 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:03.879561 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:03.879573 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:03.881552 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
W20260620 04:31:03.881814 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:03.876321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44997 (local address 127.0.98.62:35355)
0620 04:31:03.876537 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:03.876541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:03.876554 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:31:03.876662 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:03.876664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:03.876703 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:03.876775 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:03.877205 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.877725 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.878586 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:03.878828 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:03.878928 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:03.881698 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44997: BlockingRecv error: recv got EOF from 127.0.98.1:44997 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
Jun 20 04:31:04 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:04 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:04.066155 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:04.060674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:49221)
0620 04:31:04.060981 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:04.061024 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:04.061138 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:04.061415 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:04.061418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:04.061605 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:04.061699 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:04.061705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.062492 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.062495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:04.063160 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:04.063169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.063621 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.063625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:04.063701 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:04.064200 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:04.064211 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:04.066037 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
W20260620 04:31:04.066303 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:04.060896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49221 (local address 127.0.98.62:35355)
0620 04:31:04.061061 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:04.061065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:04.061082 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:31:04.061254 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:04.061257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:04.061301 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:04.061371 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:04.061827 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.062379 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.063274 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.063503 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.063600 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:04.066188 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49221: BlockingRecv error: recv got EOF from 127.0.98.2:49221 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 04:31:04 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:04 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:04.279533 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:04.274166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:46137)
0620 04:31:04.274455 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:04.274468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:04.274585 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:04.274877 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:04.274880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:04.275054 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:04.275151 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:04.275157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.275968 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.275971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:04.276544 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:04.276552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.277088 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.277091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:04.277161 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:04.277710 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:04.277722 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:04.279416 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
W20260620 04:31:04.279714 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:04.274288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46137 (local address 127.0.98.62:35355)
0620 04:31:04.274443 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:04.274446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:04.274569 (+ 123us) server_negotiation.cc:407] Connection header received
0620 04:31:04.274704 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:04.274707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:04.274765 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:04.274849 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:04.275292 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.275850 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.276700 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.276923 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.277035 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:04.279594 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46137: BlockingRecv error: recv got EOF from 127.0.98.3:46137 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 04:31:04 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:04 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:04.893733 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:04.882689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:39447)
0620 04:31:04.882997 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:04.883015 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:04.883148 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:04.883419 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:04.883422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:04.883651 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:04.883789 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:04.883798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.884756 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.884760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:04.885401 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:04.885411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.885869 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.885872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:04.885937 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:04.891589 (+ 5652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:04.891604 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:04.893576 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
W20260620 04:31:04.893986 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:04.882817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39447 (local address 127.0.98.62:35355)
0620 04:31:04.883100 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:04.883103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:04.883119 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:31:04.883242 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:04.883244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:04.883287 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:04.883365 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:04.883912 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.884646 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.885536 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:04.885766 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:04.885844 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:04.893765 (+ 7921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39447: BlockingRecv error: recv got EOF from 127.0.98.1:39447 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":63,"threads_started":1}
Jun 20 04:31:05 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:05 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:05.072611 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:05.067015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:35881)
0620 04:31:05.067394 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:05.067409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:05.067524 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:05.067760 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:05.067763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:05.067964 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:05.068064 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:05.068070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.068813 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.068817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:05.069440 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:05.069449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.069901 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.069904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:05.069994 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:05.070533 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:05.070544 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:05.072416 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":95,"threads_started":1}
W20260620 04:31:05.072691 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:05.067128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35881 (local address 127.0.98.62:35355)
0620 04:31:05.067314 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:05.067318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:05.067484 (+ 166us) server_negotiation.cc:407] Connection header received
0620 04:31:05.067607 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:05.067610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:05.067655 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:05.067721 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:05.068178 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.068700 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.069564 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.069794 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.069883 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:05.072585 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35881: BlockingRecv error: recv got EOF from 127.0.98.2:35881 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 04:31:05 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:05 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:05.285744 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:05.280271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.3:56747)
0620 04:31:05.280578 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:05.280592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:05.280691 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:05.281023 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:05.281026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:05.281209 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:05.281309 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:05.281315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.282063 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.282066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:05.282682 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:05.282692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.283173 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.283176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:05.283242 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:05.283810 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:05.283823 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:05.285593 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
W20260620 04:31:05.285928 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:05.280376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56747 (local address 127.0.98.62:35355)
0620 04:31:05.280534 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:05.280538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:05.280644 (+ 106us) server_negotiation.cc:407] Connection header received
0620 04:31:05.280795 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:05.280798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:05.280845 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:05.280921 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:05.281406 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.281938 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.282829 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.283066 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.283154 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:05.285787 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56747: BlockingRecv error: recv got EOF from 127.0.98.3:56747 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 04:31:05 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:05 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:05.900640 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:05.894727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.1:60793)
0620 04:31:05.895002 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:05.895016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:05.895114 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:05.895357 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:05.895359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:05.895541 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:05.895643 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:05.895649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.896636 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.896639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:05.897489 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:05.897504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.898058 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.898061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:05.898127 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:05.898658 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:05.898670 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:05.900524 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
W20260620 04:31:05.900815 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:05.894842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60793 (local address 127.0.98.62:35355)
0620 04:31:05.895107 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:05.895110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:05.895126 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:31:05.895196 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:05.895198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:05.895241 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:05.895314 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:05.895813 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.896474 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.897649 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:05.897890 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:05.897974 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:05.900677 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60793: BlockingRecv error: recv got EOF from 127.0.98.1:60793 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
Jun 20 04:31:06 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 04:31:06 dist-test-slave-rzqm krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 04:31:06.079758 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:06.073734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35355 (local address 127.0.98.2:60547)
0620 04:31:06.074004 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:06.074018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:06.074114 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:06.074456 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:06.074458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:06.074648 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:06.074800 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:06.074808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.075634 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.075637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:06.076287 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:06.076298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.076848 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.076852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:06.076938 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:06.077572 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:06.077587 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:06.079623 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
W20260620 04:31:06.079902 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:06.073908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60547 (local address 127.0.98.62:35355)
0620 04:31:06.074079 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:06.074082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:06.074098 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:31:06.074244 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:06.074248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:06.074300 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:06.074379 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:06.074949 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.075506 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.076442 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.076716 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.076820 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:06.079783 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60547: BlockingRecv error: recv got EOF from 127.0.98.2:60547 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 04:31:06.171834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 9129
I20260620 04:31:06.177179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 9281
I20260620 04:31:06.181910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 9413
I20260620 04:31:06.186718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 9059
2026-06-20T04:31:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17883 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T04:31:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:31:06Z Disabled control of system clock
I20260620 04:31:06.216059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34461
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 04:31:06.328070 9609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:06.328399 9609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:06.328437 9609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:06.332499 9609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:31:06.332564 9609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:06.332580 9609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:06.332613 9609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:31:06.332628 9609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:31:06.337528 9609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:06.338656 9609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:06.340070 9609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:06.349267 9614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:06.349444 9609 server_base.cc:1034] running on GCE node
W20260620 04:31:06.349267 9615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:06.349380 9617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:06.350261 9609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:06.351426 9609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:06.352626 9609 hybrid_clock.cc:648] HybridClock initialized: now 1781929866352594 us; error 49 us; skew 500 ppm
I20260620 04:31:06.354773 9609 webserver.cc:460] Webserver started at http://127.0.98.62:35865/ using document root <none> and password file <none>
I20260620 04:31:06.355376 9609 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:06.355448 9609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:06.355664 9609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:06.357895 9609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "aaab12423df54e30a8159860a8291546"
format_stamp: "Formatted at 2026-06-20 04:31:06 on dist-test-slave-rzqm"
server_key: "18139da1a4769d62d5c93eb048dc4890"
server_key_iv: "e7524978c83e499df1442b0b08611db0"
server_key_version: "encryptionkey@0"
I20260620 04:31:06.358471 9609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "aaab12423df54e30a8159860a8291546"
format_stamp: "Formatted at 2026-06-20 04:31:06 on dist-test-slave-rzqm"
server_key: "18139da1a4769d62d5c93eb048dc4890"
server_key_iv: "e7524978c83e499df1442b0b08611db0"
server_key_version: "encryptionkey@0"
I20260620 04:31:06.362493 9609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:31:06.365237 9623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:06.366664 9609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:31:06.366824 9609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "aaab12423df54e30a8159860a8291546"
format_stamp: "Formatted at 2026-06-20 04:31:06 on dist-test-slave-rzqm"
server_key: "18139da1a4769d62d5c93eb048dc4890"
server_key_iv: "e7524978c83e499df1442b0b08611db0"
server_key_version: "encryptionkey@0"
I20260620 04:31:06.366947 9609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:06.383301 9609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:06.384052 9609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:06.384233 9609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:06.391474 9609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34461
I20260620 04:31:06.391526 9675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34461 every 8 connection(s)
I20260620 04:31:06.392460 9609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:31:06.395768 9676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:31:06.401758 9676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546: Bootstrap starting.
I20260620 04:31:06.402148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 9609
I20260620 04:31:06.402259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:31:06.402550 392 external_mini_cluster.cc:1428] Setting key 3239b78b8e5cb748ffe3149a62f662ba
I20260620 04:31:06.404677 9676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546: Neither blocks nor log segments found. Creating new log.
I20260620 04:31:06.405510 9676 log.cc:824] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546: Log is configured to *not* fsync() on all Append() calls
I20260620 04:31:06.408192 9676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546: No bootstrap required, opened a new log
I20260620 04:31:06.413061 9676 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaab12423df54e30a8159860a8291546" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } }
I20260620 04:31:06.413408 9676 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:31:06.413532 9676 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaab12423df54e30a8159860a8291546, State: Initialized, Role: FOLLOWER
I20260620 04:31:06.414167 9676 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaab12423df54e30a8159860a8291546" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } }
I20260620 04:31:06.414355 9676 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:31:06.414368 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:06.404052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.0.98.62:34461)
0620 04:31:06.404780 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:06.404800 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:06.404850 (+ 50us) server_negotiation.cc:407] Connection header received
0620 04:31:06.405565 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:06.405585 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:06.406112 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:06.406628 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:06.407180 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.408191 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.409591 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.410193 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.410363 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:06.411464 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:06.411472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:06.411492 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:31:06.412023 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:06.412107 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:31:06.412137 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:06.412242 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:06.412564 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 04:31:06.412976 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":156,"threads_started":1}
I20260620 04:31:06.414448 9676 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:31:06.414675 9676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:31:06.415993 9676 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaab12423df54e30a8159860a8291546" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } }
I20260620 04:31:06.416409 9676 leader_election.cc:302] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaab12423df54e30a8159860a8291546; no voters:
I20260620 04:31:06.416777 9676 leader_election.cc:288] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:31:06.416942 9681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:31:06.417469 9681 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [term 1 LEADER]: Becoming Leader. State: Replica: aaab12423df54e30a8159860a8291546, State: Running, Role: LEADER
I20260620 04:31:06.418069 9681 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaab12423df54e30a8159860a8291546" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } }
I20260620 04:31:06.418211 9676 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:31:06.420128 9682 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaab12423df54e30a8159860a8291546. Latest consensus state: current_term: 1 leader_uuid: "aaab12423df54e30a8159860a8291546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaab12423df54e30a8159860a8291546" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } }
I20260620 04:31:06.420265 9682 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [sys.catalog]: This master's current role is: LEADER
I20260620 04:31:06.420089 9683 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaab12423df54e30a8159860a8291546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaab12423df54e30a8159860a8291546" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34461 } } }
I20260620 04:31:06.420691 9683 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546 [sys.catalog]: This master's current role is: LEADER
W20260620 04:31:06.423516 9695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:31:06.423591 9695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:31:06.423660 9696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:31:06.424521 9696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:31:06.430846 9696 catalog_manager.cc:1347] Generated new cluster ID: ef8ab56e7671491c9da9ba22837bfa03
I20260620 04:31:06.430923 9696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:31:06.491773 9696 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:31:06.493122 9696 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:31:06.501274 9696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaab12423df54e30a8159860a8291546: Generated new TSK 0
I20260620 04:31:06.502256 9696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:31:06.516001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--builtin_ntp_servers=127.0.98.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 04:31:06.630437 9700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:06.630748 9700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:06.630784 9700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:06.634972 9700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:06.635038 9700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:06.635097 9700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:31:06.640223 9700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:06.641386 9700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:06.642763 9700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:06.652323 9705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:06.652341 9706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:06.652333 9708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:06.652525 9700 server_base.cc:1034] running on GCE node
I20260620 04:31:06.653100 9700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:06.653782 9700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:06.654953 9700 hybrid_clock.cc:648] HybridClock initialized: now 1781929866654932 us; error 34 us; skew 500 ppm
I20260620 04:31:06.656936 9700 webserver.cc:460] Webserver started at http://127.0.98.1:35463/ using document root <none> and password file <none>
I20260620 04:31:06.657552 9700 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:06.657618 9700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:06.657788 9700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:06.659945 9700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "47739ced665d43b4b5dc71f73b6ed4f9"
format_stamp: "Formatted at 2026-06-20 04:31:06 on dist-test-slave-rzqm"
server_key: "11ed1b608a0bf151182bce6d769a5c20"
server_key_iv: "3920ea6572908cc96109d0bf8ea97292"
server_key_version: "encryptionkey@0"
I20260620 04:31:06.660465 9700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47739ced665d43b4b5dc71f73b6ed4f9"
format_stamp: "Formatted at 2026-06-20 04:31:06 on dist-test-slave-rzqm"
server_key: "11ed1b608a0bf151182bce6d769a5c20"
server_key_iv: "3920ea6572908cc96109d0bf8ea97292"
server_key_version: "encryptionkey@0"
I20260620 04:31:06.664317 9700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:31:06.667152 9714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:06.668496 9700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:31:06.668609 9700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "47739ced665d43b4b5dc71f73b6ed4f9"
format_stamp: "Formatted at 2026-06-20 04:31:06 on dist-test-slave-rzqm"
server_key: "11ed1b608a0bf151182bce6d769a5c20"
server_key_iv: "3920ea6572908cc96109d0bf8ea97292"
server_key_version: "encryptionkey@0"
I20260620 04:31:06.668738 9700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:06.685163 9700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:06.685933 9700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:06.686124 9700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:06.686735 9700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:06.687716 9700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:06.687762 9700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:06.687801 9700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:06.687817 9700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:06.696462 9700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40333
I20260620 04:31:06.696579 9827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40333 every 8 connection(s)
I20260620 04:31:06.697484 9700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:31:06.703306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 9700
I20260620 04:31:06.703414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:31:06.703677 392 external_mini_cluster.cc:1428] Setting key 3bc7314aa021db7b3201e4475cb0760a
I20260620 04:31:06.705862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--builtin_ntp_servers=127.0.98.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 04:31:06.707381 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:06.699334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60167 (local address 127.0.98.62:34461)
0620 04:31:06.699479 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:06.699483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:06.700652 (+ 1169us) server_negotiation.cc:407] Connection header received
0620 04:31:06.702095 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:06.702098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:06.702147 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:06.702225 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:06.703182 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.703759 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.705372 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.705678 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.705785 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:06.706861 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:06.706866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:06.706868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:31:06.706883 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:06.706908 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:31:06.706919 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:06.706995 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:06.707163 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 04:31:06.707215 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:31:06.708443 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:06.699809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.1:60167)
0620 04:31:06.700502 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:06.700550 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:06.701860 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:06.702401 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:06.702408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:06.702496 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:06.703025 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:06.703045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.703882 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.703885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:06.705266 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:06.705279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.705819 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.705821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:06.705975 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:06.706595 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:31:06.706597 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:06.706638 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:31:06.706706 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:06.707017 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:06.707025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:06.707031 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:31:06.707165 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 04:31:06.707480 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":128,"threads_started":1}
I20260620 04:31:06.709913 9828 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34461
I20260620 04:31:06.710119 9828 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:06.710669 9828 heartbeater.cc:511] Master 127.0.98.62:34461 requested a full tablet report, sending...
I20260620 04:31:06.712493 9640 ts_manager.cc:194] Registered new tserver with Master: 47739ced665d43b4b5dc71f73b6ed4f9 (127.0.98.1:40333)
I20260620 04:31:06.714162 9640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60167
W20260620 04:31:06.819600 9831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:06.819907 9831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:06.819947 9831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:06.824100 9831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:06.824178 9831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:06.824234 9831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:31:06.829169 9831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:06.830255 9831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:06.831579 9831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:06.841259 9836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:06.841269 9839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:06.841348 9831 server_base.cc:1034] running on GCE node
W20260620 04:31:06.841274 9837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:06.841952 9831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:06.842653 9831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:06.843832 9831 hybrid_clock.cc:648] HybridClock initialized: now 1781929866843809 us; error 40 us; skew 500 ppm
I20260620 04:31:06.845789 9831 webserver.cc:460] Webserver started at http://127.0.98.2:45095/ using document root <none> and password file <none>
I20260620 04:31:06.846436 9831 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:06.846510 9831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:06.846719 9831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:06.848927 9831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "b7900b3c6fcd41498977967f43016da9"
format_stamp: "Formatted at 2026-06-20 04:31:06 on dist-test-slave-rzqm"
server_key: "4ccb652166f2daa0339692418f5c97ae"
server_key_iv: "781e8182f194a90edc171a1d0d903f4f"
server_key_version: "encryptionkey@0"
I20260620 04:31:06.849624 9831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b7900b3c6fcd41498977967f43016da9"
format_stamp: "Formatted at 2026-06-20 04:31:06 on dist-test-slave-rzqm"
server_key: "4ccb652166f2daa0339692418f5c97ae"
server_key_iv: "781e8182f194a90edc171a1d0d903f4f"
server_key_version: "encryptionkey@0"
I20260620 04:31:06.853459 9831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:31:06.856184 9845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:06.857626 9831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:31:06.857749 9831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "b7900b3c6fcd41498977967f43016da9"
format_stamp: "Formatted at 2026-06-20 04:31:06 on dist-test-slave-rzqm"
server_key: "4ccb652166f2daa0339692418f5c97ae"
server_key_iv: "781e8182f194a90edc171a1d0d903f4f"
server_key_version: "encryptionkey@0"
I20260620 04:31:06.857857 9831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:06.886070 9831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:06.886821 9831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:06.887007 9831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:06.887606 9831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:06.888583 9831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:06.888631 9831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:06.888684 9831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:06.888710 9831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:06.897142 9831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38495
I20260620 04:31:06.897181 9958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38495 every 8 connection(s)
I20260620 04:31:06.898083 9831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:31:06.902900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 9831
I20260620 04:31:06.903025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:31:06.903297 392 external_mini_cluster.cc:1428] Setting key 66e14f0b4cd8f08a19bcb86ba576bd84
I20260620 04:31:06.905184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--builtin_ntp_servers=127.0.98.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 04:31:06.907797 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:06.900002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56609 (local address 127.0.98.62:34461)
0620 04:31:06.900143 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:06.900147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:06.901316 (+ 1169us) server_negotiation.cc:407] Connection header received
0620 04:31:06.902733 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:06.902736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:06.902794 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:06.902890 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:06.903856 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.904428 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.905694 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.905993 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.906097 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:06.907307 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:06.907310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:06.907312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:31:06.907325 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:06.907345 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:31:06.907353 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:06.907413 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:06.907592 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 04:31:06.907655 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:31:06.908736 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:06.900444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.2:56609)
0620 04:31:06.901180 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:06.901222 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:06.902516 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:06.903066 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:06.903075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:06.903174 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:06.903743 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:06.903757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.904548 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.904552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:06.905587 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:06.905596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:06.906160 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:06.906163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:06.906338 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:06.907046 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:31:06.907048 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:06.907090 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:31:06.907160 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:06.907447 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:06.907455 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:06.907459 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:31:06.907571 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 04:31:06.907899 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20260620 04:31:06.909857 9959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34461
I20260620 04:31:06.910096 9959 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:06.910614 9959 heartbeater.cc:511] Master 127.0.98.62:34461 requested a full tablet report, sending...
I20260620 04:31:06.911660 9640 ts_manager.cc:194] Registered new tserver with Master: b7900b3c6fcd41498977967f43016da9 (127.0.98.2:38495)
I20260620 04:31:06.912236 9640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56609
W20260620 04:31:07.017246 9962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:07.017555 9962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:07.017608 9962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:07.021873 9962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:07.021934 9962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:07.022010 9962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:31:07.026798 9962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:07.027917 9962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:07.029322 9962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:07.038815 9968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:07.038837 9970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:07.038991 9962 server_base.cc:1034] running on GCE node
W20260620 04:31:07.038821 9967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:07.039429 9962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:07.040035 9962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:07.041186 9962 hybrid_clock.cc:648] HybridClock initialized: now 1781929867041151 us; error 46 us; skew 500 ppm
I20260620 04:31:07.043040 9962 webserver.cc:460] Webserver started at http://127.0.98.3:37115/ using document root <none> and password file <none>
I20260620 04:31:07.043594 9962 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:07.043658 9962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:07.043843 9962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:07.045938 9962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "a58dbb417686422e8ccb0f4569d00d8c"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "5e3d552c243f9e45e5dca6ea6495f90a"
server_key_iv: "4ce111d9850f2ea9c280c332ed902e08"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.046506 9962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a58dbb417686422e8ccb0f4569d00d8c"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "5e3d552c243f9e45e5dca6ea6495f90a"
server_key_iv: "4ce111d9850f2ea9c280c332ed902e08"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.050261 9962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:31:07.052772 9976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.054080 9962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:31:07.054195 9962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "a58dbb417686422e8ccb0f4569d00d8c"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "5e3d552c243f9e45e5dca6ea6495f90a"
server_key_iv: "4ce111d9850f2ea9c280c332ed902e08"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.054301 9962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:07.064527 9962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:07.065251 9962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:07.065420 9962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:07.065994 9962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:07.066919 9962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:07.066964 9962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.067018 9962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:07.067044 9962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.075852 9962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40671
I20260620 04:31:07.075881 10089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40671 every 8 connection(s)
I20260620 04:31:07.076740 9962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:31:07.081508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 9962
I20260620 04:31:07.081632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:31:07.081933 392 external_mini_cluster.cc:1428] Setting key 74177f060e15b46fcff68cc04ebfd320
I20260620 04:31:07.085940 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:07.078504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52267 (local address 127.0.98.62:34461)
0620 04:31:07.078668 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:07.078672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:07.079767 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 04:31:07.081108 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:07.081111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:07.081158 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:07.081232 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:07.082436 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:07.082965 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:07.084115 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:07.084338 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:07.084425 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:07.085486 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:07.085490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:07.085492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:31:07.085506 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:07.085526 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:31:07.085535 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:07.085592 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:07.085727 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 04:31:07.085789 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 04:31:07.086839 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:07.078922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34461 (local address 127.0.98.3:52267)
0620 04:31:07.079620 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:07.079662 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:07.080886 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:07.081468 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:07.081477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:07.081591 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:07.082280 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:07.082298 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:07.083094 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:07.083097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:07.084002 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:07.084011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:07.084485 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:07.084490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:07.084646 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:07.085213 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:31:07.085214 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:07.085256 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:31:07.085325 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:07.085601 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:07.085610 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:07.085616 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:31:07.085723 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 04:31:07.086020 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260620 04:31:07.087890 10090 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34461
I20260620 04:31:07.088106 10090 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:07.088640 10090 heartbeater.cc:511] Master 127.0.98.62:34461 requested a full tablet report, sending...
I20260620 04:31:07.089607 9640 ts_manager.cc:194] Registered new tserver with Master: a58dbb417686422e8ccb0f4569d00d8c (127.0.98.3:40671)
I20260620 04:31:07.090142 9640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52267
I20260620 04:31:07.095700 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 04:31:07.098619 9679 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:07.097657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.0.98.62:34461)
0620 04:31:07.097839 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:07.097843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:07.097947 (+ 104us) server_negotiation.cc:407] Connection header received
0620 04:31:07.098096 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:07.098099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:07.098139 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:07.098200 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:07.098515 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46858: BlockingRecv error: recv got EOF from 127.0.0.1:46858 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 04:31:07.099059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 9700
I20260620 04:31:07.103765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 9831
I20260620 04:31:07.108026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 9962
I20260620 04:31:07.112577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 9609
2026-06-20T04:31:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (924 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T04:31:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:31:07Z Disabled control of system clock
I20260620 04:31:07.140151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45021
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 04:31:07.251664 10101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:07.251953 10101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:07.251988 10101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:07.256085 10101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:31:07.256146 10101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:07.256166 10101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:07.256182 10101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:31:07.256197 10101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:31:07.261065 10101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45021
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:07.262208 10101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:07.263880 10101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:07.272676 10109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:07.272676 10107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:07.272773 10101 server_base.cc:1034] running on GCE node
W20260620 04:31:07.272681 10106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:07.273500 10101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:07.274636 10101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:07.275805 10101 hybrid_clock.cc:648] HybridClock initialized: now 1781929867275776 us; error 38 us; skew 500 ppm
I20260620 04:31:07.277812 10101 webserver.cc:460] Webserver started at http://127.0.98.62:46145/ using document root <none> and password file <none>
I20260620 04:31:07.278398 10101 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:07.278468 10101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:07.278676 10101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:07.280838 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "a689e1814ac74b44aa70e695f40ad1a2"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "30e22e28cffb068ff158720ec91d92a0"
server_key_iv: "13a6537a801cd576b82461f8876b68a3"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.281461 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "a689e1814ac74b44aa70e695f40ad1a2"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "30e22e28cffb068ff158720ec91d92a0"
server_key_iv: "13a6537a801cd576b82461f8876b68a3"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.285377 10101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:31:07.288014 10115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.289431 10101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:31:07.289559 10101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "a689e1814ac74b44aa70e695f40ad1a2"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "30e22e28cffb068ff158720ec91d92a0"
server_key_iv: "13a6537a801cd576b82461f8876b68a3"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.289675 10101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:07.294325 10101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:07.294920 10101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:07.295078 10101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:07.301755 10101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45021
I20260620 04:31:07.301827 10167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45021 every 8 connection(s)
I20260620 04:31:07.302748 10101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:31:07.305879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 10101
I20260620 04:31:07.305976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:31:07.306245 392 external_mini_cluster.cc:1428] Setting key 1ac80402e5d12ca5db725824e337b88a
I20260620 04:31:07.306236 10168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:31:07.311976 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:07.307587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.0.98.62:45021)
0620 04:31:07.308218 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:07.308235 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:07.308270 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:31:07.308949 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:07.308968 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:07.309392 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:07.309692 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:07.310532 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:07.310537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:07.310552 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:31:07.310571 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:07.310647 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:31:07.310665 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:07.310787 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:07.310954 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 04:31:07.311229 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260620 04:31:07.313354 10168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2: Bootstrap starting.
I20260620 04:31:07.315543 10168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2: Neither blocks nor log segments found. Creating new log.
I20260620 04:31:07.316277 10168 log.cc:824] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2: Log is configured to *not* fsync() on all Append() calls
I20260620 04:31:07.318169 10168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2: No bootstrap required, opened a new log
I20260620 04:31:07.321429 10168 raft_consensus.cc:348] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a689e1814ac74b44aa70e695f40ad1a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } }
I20260620 04:31:07.321645 10168 raft_consensus.cc:374] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:31:07.321679 10168 raft_consensus.cc:729] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a689e1814ac74b44aa70e695f40ad1a2, State: Initialized, Role: FOLLOWER
I20260620 04:31:07.322038 10168 consensus_queue.cc:260] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a689e1814ac74b44aa70e695f40ad1a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } }
I20260620 04:31:07.322144 10168 raft_consensus.cc:388] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:31:07.322180 10168 raft_consensus.cc:482] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:31:07.322256 10168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:31:07.323076 10168 raft_consensus.cc:504] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a689e1814ac74b44aa70e695f40ad1a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } }
I20260620 04:31:07.323378 10168 leader_election.cc:302] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a689e1814ac74b44aa70e695f40ad1a2; no voters:
I20260620 04:31:07.323681 10168 leader_election.cc:288] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:31:07.323817 10173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:31:07.324030 10173 raft_consensus.cc:686] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [term 1 LEADER]: Becoming Leader. State: Replica: a689e1814ac74b44aa70e695f40ad1a2, State: Running, Role: LEADER
I20260620 04:31:07.324332 10173 consensus_queue.cc:237] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a689e1814ac74b44aa70e695f40ad1a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } }
I20260620 04:31:07.324872 10168 sys_catalog.cc:564] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:31:07.326020 10175 sys_catalog.cc:455] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a689e1814ac74b44aa70e695f40ad1a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a689e1814ac74b44aa70e695f40ad1a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } } }
I20260620 04:31:07.326028 10174 sys_catalog.cc:455] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a689e1814ac74b44aa70e695f40ad1a2. Latest consensus state: current_term: 1 leader_uuid: "a689e1814ac74b44aa70e695f40ad1a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a689e1814ac74b44aa70e695f40ad1a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } } }
I20260620 04:31:07.326174 10175 sys_catalog.cc:458] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [sys.catalog]: This master's current role is: LEADER
I20260620 04:31:07.326175 10174 sys_catalog.cc:458] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2 [sys.catalog]: This master's current role is: LEADER
I20260620 04:31:07.326524 10182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:31:07.329110 10182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:31:07.334005 10182 catalog_manager.cc:1347] Generated new cluster ID: cd4f43a20bf34513a0b0620a70cad84e
I20260620 04:31:07.334071 10182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:31:07.340924 10182 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:31:07.341279 10182 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:31:07.346944 10182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a689e1814ac74b44aa70e695f40ad1a2: Generated new TSK 0
I20260620 04:31:07.347612 10182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:31:07.354421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--builtin_ntp_servers=127.0.98.20:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 04:31:07.465932 10192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:07.466238 10192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:07.466295 10192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:07.470424 10192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:07.470484 10192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:07.470566 10192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:31:07.475457 10192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:07.476588 10192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:07.477984 10192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:07.487357 10200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:07.487362 10197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:07.487362 10198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:07.487566 10192 server_base.cc:1034] running on GCE node
I20260620 04:31:07.487998 10192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:07.488627 10192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:07.489794 10192 hybrid_clock.cc:648] HybridClock initialized: now 1781929867489774 us; error 32 us; skew 500 ppm
I20260620 04:31:07.491617 10192 webserver.cc:460] Webserver started at http://127.0.98.1:45367/ using document root <none> and password file <none>
I20260620 04:31:07.492166 10192 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:07.492224 10192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:07.492384 10192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:07.494431 10192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "bdeec7966c5b40eaa5e0c16a137b2b89"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "cacc3e1ab7b1cf98c07ac48d7ddbc65a"
server_key_iv: "b5211588614c3ba4fef9e208f3d13a3b"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.494940 10192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bdeec7966c5b40eaa5e0c16a137b2b89"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "cacc3e1ab7b1cf98c07ac48d7ddbc65a"
server_key_iv: "b5211588614c3ba4fef9e208f3d13a3b"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.498590 10192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:31:07.500906 10206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.502132 10192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:31:07.502233 10192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "bdeec7966c5b40eaa5e0c16a137b2b89"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "cacc3e1ab7b1cf98c07ac48d7ddbc65a"
server_key_iv: "b5211588614c3ba4fef9e208f3d13a3b"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.502316 10192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:07.506594 10192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:07.507143 10192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:07.507280 10192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:07.507754 10192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:07.508597 10192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:07.508642 10192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.508677 10192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:07.508692 10192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.517268 10192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33685
I20260620 04:31:07.517300 10319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33685 every 8 connection(s)
I20260620 04:31:07.518200 10192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:31:07.520910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 10192
I20260620 04:31:07.521046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:31:07.521317 392 external_mini_cluster.cc:1428] Setting key e0e614309d9be5b2ea50eea757f1ec70
I20260620 04:31:07.523077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--builtin_ntp_servers=127.0.98.20:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 04:31:07.524163 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:07.519911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47535 (local address 127.0.98.62:45021)
0620 04:31:07.520028 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:07.520031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:07.521233 (+ 1202us) server_negotiation.cc:407] Connection header received
0620 04:31:07.522550 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:07.522553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:07.522594 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:07.522664 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:07.523738 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:07.523742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:07.523745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:31:07.523757 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:07.523780 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:31:07.523788 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:07.523860 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:07.523991 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 04:31:07.524033 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:31:07.525125 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:07.520329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45021 (local address 127.0.98.1:47535)
0620 04:31:07.520985 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:07.521125 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:07.522331 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:07.522832 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:07.522840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:07.522933 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:07.523478 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:31:07.523480 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:07.523520 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:31:07.523593 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:07.523856 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:07.523865 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:07.523871 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:31:07.524027 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 04:31:07.524283 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260620 04:31:07.526432 10320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45021
I20260620 04:31:07.526669 10320 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:07.527114 10320 heartbeater.cc:511] Master 127.0.98.62:45021 requested a full tablet report, sending...
I20260620 04:31:07.528803 10132 ts_manager.cc:194] Registered new tserver with Master: bdeec7966c5b40eaa5e0c16a137b2b89 (127.0.98.1:33685)
W20260620 04:31:07.637208 10323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:07.637488 10323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:07.637532 10323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:07.641606 10323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:07.641662 10323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:07.641719 10323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:31:07.646493 10323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:07.647536 10323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:07.648891 10323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:07.658581 10331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:07.658689 10329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:07.658689 10328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:07.658872 10323 server_base.cc:1034] running on GCE node
I20260620 04:31:07.659335 10323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:07.659929 10323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:07.661079 10323 hybrid_clock.cc:648] HybridClock initialized: now 1781929867661046 us; error 32 us; skew 500 ppm
I20260620 04:31:07.662889 10323 webserver.cc:460] Webserver started at http://127.0.98.2:41397/ using document root <none> and password file <none>
I20260620 04:31:07.663451 10323 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:07.663525 10323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:07.663730 10323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:07.665903 10323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "b25edf38ae8b4f9eaf40d8ae8c98e57e"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "6ff45ef321ed5c49089666f5f1315c4b"
server_key_iv: "317e28c2de7052d0ca496bfda3559dc0"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.666472 10323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b25edf38ae8b4f9eaf40d8ae8c98e57e"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "6ff45ef321ed5c49089666f5f1315c4b"
server_key_iv: "317e28c2de7052d0ca496bfda3559dc0"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.670389 10323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:31:07.672809 10337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.674181 10323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:31:07.674299 10323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "b25edf38ae8b4f9eaf40d8ae8c98e57e"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "6ff45ef321ed5c49089666f5f1315c4b"
server_key_iv: "317e28c2de7052d0ca496bfda3559dc0"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.674400 10323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:07.678948 10323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:07.679580 10323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:07.679736 10323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:07.680270 10323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:07.681202 10323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:07.681250 10323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.681300 10323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:07.681327 10323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.689525 10323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38897
I20260620 04:31:07.689567 10450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38897 every 8 connection(s)
I20260620 04:31:07.690431 10323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:31:07.696429 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:07.692350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56993 (local address 127.0.98.62:45021)
0620 04:31:07.692513 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:07.692517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:07.693661 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 04:31:07.694927 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:07.694930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:07.694967 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:07.695057 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:07.696083 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:07.696086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:07.696088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:31:07.696099 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:07.696117 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:31:07.696124 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:07.696204 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:07.696290 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 04:31:07.696323 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:31:07.697364 10453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:07.692728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45021 (local address 127.0.98.2:56993)
0620 04:31:07.693513 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:07.693566 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:07.694705 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:07.695205 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:07.695211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:07.695302 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:07.695801 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:31:07.695803 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:07.695842 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:31:07.695909 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:07.696194 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:07.696198 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:07.696202 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:31:07.696321 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 04:31:07.696549 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260620 04:31:07.698444 10451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45021
I20260620 04:31:07.698659 10451 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:07.699080 10451 heartbeater.cc:511] Master 127.0.98.62:45021 requested a full tablet report, sending...
I20260620 04:31:07.699652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 10323
I20260620 04:31:07.699741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:31:07.700023 392 external_mini_cluster.cc:1428] Setting key 45de74d90bc7766322bc4cdfdb1b7661
I20260620 04:31:07.700233 10132 ts_manager.cc:194] Registered new tserver with Master: b25edf38ae8b4f9eaf40d8ae8c98e57e (127.0.98.2:38897)
I20260620 04:31:07.701804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--builtin_ntp_servers=127.0.98.20:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 04:31:07.814981 10454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:07.815270 10454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:07.815311 10454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:07.819603 10454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:07.819677 10454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:07.819737 10454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:31:07.824729 10454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:07.826094 10454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:07.827534 10454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:07.837634 10462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:07.837740 10460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:07.837667 10459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:07.837651 10454 server_base.cc:1034] running on GCE node
I20260620 04:31:07.838310 10454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:07.838965 10454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:07.840138 10454 hybrid_clock.cc:648] HybridClock initialized: now 1781929867840115 us; error 35 us; skew 500 ppm
I20260620 04:31:07.842183 10454 webserver.cc:460] Webserver started at http://127.0.98.3:43497/ using document root <none> and password file <none>
I20260620 04:31:07.843036 10454 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:07.843120 10454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:07.843327 10454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:07.845640 10454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "d220d5c8529b441ba7bb155cc4cd7270"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "fac7d172099d93bf93d17b00ec2964ba"
server_key_iv: "d71b501fca5af2755d84f3aa1c31f736"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.846238 10454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d220d5c8529b441ba7bb155cc4cd7270"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "fac7d172099d93bf93d17b00ec2964ba"
server_key_iv: "d71b501fca5af2755d84f3aa1c31f736"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.850178 10454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 04:31:07.853058 10468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.854451 10454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:31:07.854593 10454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "d220d5c8529b441ba7bb155cc4cd7270"
format_stamp: "Formatted at 2026-06-20 04:31:07 on dist-test-slave-rzqm"
server_key: "fac7d172099d93bf93d17b00ec2964ba"
server_key_iv: "d71b501fca5af2755d84f3aa1c31f736"
server_key_version: "encryptionkey@0"
I20260620 04:31:07.854724 10454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:07.859488 10454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:07.860117 10454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:07.860275 10454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:07.860829 10454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:07.861779 10454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:07.861825 10454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.861863 10454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:07.861892 10454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:07.870078 10454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35051
I20260620 04:31:07.870132 10581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35051 every 8 connection(s)
I20260620 04:31:07.870986 10454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:31:07.876868 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:07.872783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49345 (local address 127.0.98.62:45021)
0620 04:31:07.872939 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:07.872944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:07.874062 (+ 1118us) server_negotiation.cc:407] Connection header received
0620 04:31:07.875301 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:07.875304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:07.875340 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:07.875410 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:07.876512 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:07.876515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:07.876518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:31:07.876531 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:07.876550 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:31:07.876558 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:07.876632 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:07.876724 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 04:31:07.876768 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:31:07.877835 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:07.873250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45021 (local address 127.0.98.3:49345)
0620 04:31:07.873914 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:07.873954 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:07.875120 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:07.875584 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:07.875591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:07.875679 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:07.876262 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:31:07.876263 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:07.876304 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:31:07.876375 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:07.876624 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:07.876629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:07.876633 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 04:31:07.876753 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 04:31:07.876957 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260620 04:31:07.877887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 10454
I20260620 04:31:07.877974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:31:07.878298 392 external_mini_cluster.cc:1428] Setting key d0edfb5823b7b995b9fb512ac6034e90
I20260620 04:31:07.878940 10582 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45021
I20260620 04:31:07.879179 10582 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:07.879603 10582 heartbeater.cc:511] Master 127.0.98.62:45021 requested a full tablet report, sending...
I20260620 04:31:07.880530 10132 ts_manager.cc:194] Registered new tserver with Master: d220d5c8529b441ba7bb155cc4cd7270 (127.0.98.3:35051)
I20260620 04:31:07.892473 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 04:31:07.895424 10171 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:07.894596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.0.98.62:45021)
0620 04:31:07.894727 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:07.894731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:07.894915 (+ 184us) server_negotiation.cc:407] Connection header received
0620 04:31:07.895001 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:07.895004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:07.895041 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:07.895113 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:07.895300 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44564: BlockingRecv error: recv got EOF from 127.0.0.1:44564 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:31:07.895622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 10192
I20260620 04:31:07.900559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 10323
I20260620 04:31:07.905254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 10454
I20260620 04:31:07.909672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 10101
2026-06-20T04:31:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (796 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:31:07 dist-test-slave-rzqm krb5kdc[10592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:31:07 dist-test-slave-rzqm krb5kdc[10592](info): set up 2 sockets
Jun 20 04:31:07 dist-test-slave-rzqm krb5kdc[10592](info): commencing operation
krb5kdc: starting...
W20260620 04:31:08.865888 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:31:08 dist-test-slave-rzqm krb5kdc[10592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:31:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:31:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:31:08.992233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43573
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:31:09.104067 10605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:09.104367 10605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:09.104410 10605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:09.108560 10605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:31:09.108654 10605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:09.108677 10605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:09.108693 10605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:31:09.108707 10605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:31:09.114336 10605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:09.115440 10605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:09.116545 10605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:09.125383 10611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:09.125710 10605 server_base.cc:1034] running on GCE node
W20260620 04:31:09.125383 10610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:09.125386 10613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:09.126384 10605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:09.127681 10605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:09.128902 10605 hybrid_clock.cc:648] HybridClock initialized: now 1781929869128887 us; error 43 us; skew 500 ppm
Jun 20 04:31:09 dist-test-slave-rzqm krb5kdc[10592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:31:09.131882 10605 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:31:09.132836 10605 webserver.cc:460] Webserver started at http://127.0.98.62:34793/ using document root <none> and password file <none>
I20260620 04:31:09.133442 10605 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:09.133507 10605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:09.133677 10605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:09.135807 10605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "9d26ca13196a4d348b3033468621bba3"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "a25f07700976107508aad3637c79a64e"
server_key_iv: "bc644685b06e98af907b770b0bcf4988"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.136376 10605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d26ca13196a4d348b3033468621bba3"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "a25f07700976107508aad3637c79a64e"
server_key_iv: "bc644685b06e98af907b770b0bcf4988"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.140384 10605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:31:09.143100 10620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:09.144654 10605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:31:09.144794 10605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "9d26ca13196a4d348b3033468621bba3"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "a25f07700976107508aad3637c79a64e"
server_key_iv: "bc644685b06e98af907b770b0bcf4988"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.144939 10605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:09.159674 10605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:09.163061 10605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:09.163241 10605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:09.170096 10605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43573
I20260620 04:31:09.170374 10672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43573 every 8 connection(s)
I20260620 04:31:09.171092 10605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:31:09.173904 10673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:31:09.179054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 10605
I20260620 04:31:09.179159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:31:09.179409 392 external_mini_cluster.cc:1428] Setting key 88752d5a235c3a5f2280f94956538c64
I20260620 04:31:09.179535 10673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3: Bootstrap starting.
I20260620 04:31:09.182145 10673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3: Neither blocks nor log segments found. Creating new log.
I20260620 04:31:09.182948 10673 log.cc:824] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3: Log is configured to *not* fsync() on all Append() calls
I20260620 04:31:09.184932 10673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3: No bootstrap required, opened a new log
Jun 20 04:31:09 dist-test-slave-rzqm krb5kdc[10592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:31:09.189103 10673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d26ca13196a4d348b3033468621bba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43573 } }
I20260620 04:31:09.189371 10673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:31:09.189461 10673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d26ca13196a4d348b3033468621bba3, State: Initialized, Role: FOLLOWER
I20260620 04:31:09.189877 10673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d26ca13196a4d348b3033468621bba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43573 } }
I20260620 04:31:09.190007 10673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:31:09.190073 10673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:31:09.190168 10673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:31:09.191053 10673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d26ca13196a4d348b3033468621bba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43573 } }
I20260620 04:31:09.191447 10673 leader_election.cc:302] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d26ca13196a4d348b3033468621bba3; no voters:
I20260620 04:31:09.191797 10673 leader_election.cc:288] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:31:09.191910 10678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:31:09.192162 10678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [term 1 LEADER]: Becoming Leader. State: Replica: 9d26ca13196a4d348b3033468621bba3, State: Running, Role: LEADER
I20260620 04:31:09.192539 10678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d26ca13196a4d348b3033468621bba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43573 } }
I20260620 04:31:09.193066 10673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:31:09.194290 10679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d26ca13196a4d348b3033468621bba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d26ca13196a4d348b3033468621bba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43573 } } }
I20260620 04:31:09.194501 10679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [sys.catalog]: This master's current role is: LEADER
I20260620 04:31:09.194844 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.180794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.0.98.62:43573)
0620 04:31:09.181251 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:09.181273 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:09.181314 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:31:09.181962 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:09.181980 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:09.182332 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:09.182631 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:09.183527 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.184663 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.185825 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.186309 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.186478 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.189286 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:09.189292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:09.189305 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:31:09.189334 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:09.191147 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:09.191677 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:09.191684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:09.191691 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:09.191864 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:09.192274 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:09.192277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:09.192279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:09.192831 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:09.193067 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:09.193448 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 04:31:09.193811 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":171,"threads_started":1}
I20260620 04:31:09.195076 10688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:31:09.194859 10680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d26ca13196a4d348b3033468621bba3. Latest consensus state: current_term: 1 leader_uuid: "9d26ca13196a4d348b3033468621bba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d26ca13196a4d348b3033468621bba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43573 } } }
I20260620 04:31:09.195732 10680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3 [sys.catalog]: This master's current role is: LEADER
I20260620 04:31:09.198241 10688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:31:09.203872 10688 catalog_manager.cc:1347] Generated new cluster ID: 05634a6004cc42a29981badedd3f747b
I20260620 04:31:09.203959 10688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:31:09.232136 10688 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:31:09.233203 10688 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:31:09.250332 10688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3: Generated new TSK 0
I20260620 04:31:09.251083 10688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:31:09.289608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43573
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:31:09.401139 10699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:09.401417 10699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:09.401459 10699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:09.405611 10699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:09.405673 10699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:09.405737 10699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:31:09.411119 10699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:09.412216 10699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:09.413322 10699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:31:09.423065 10699 server_base.cc:1034] running on GCE node
W20260620 04:31:09.423014 10705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:09.423027 10704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:09.423014 10707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:09.423763 10699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:09.424461 10699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:09.425649 10699 hybrid_clock.cc:648] HybridClock initialized: now 1781929869425613 us; error 51 us; skew 500 ppm
Jun 20 04:31:09 dist-test-slave-rzqm krb5kdc[10592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:31:09.428543 10699 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:31:09.429530 10699 webserver.cc:460] Webserver started at http://127.0.98.1:39101/ using document root <none> and password file <none>
I20260620 04:31:09.430101 10699 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:09.430171 10699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:09.430382 10699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:09.432456 10699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "db23e4c03705476db27cea6dae964b26"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "784fc693af69735b49803f56b4a9cc41"
server_key_iv: "f3b83fc9f50c3348ce13debe31eff302"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.432989 10699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "db23e4c03705476db27cea6dae964b26"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "784fc693af69735b49803f56b4a9cc41"
server_key_iv: "f3b83fc9f50c3348ce13debe31eff302"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.436836 10699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 04:31:09.439327 10714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:09.440785 10699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:31:09.440897 10699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "db23e4c03705476db27cea6dae964b26"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "784fc693af69735b49803f56b4a9cc41"
server_key_iv: "f3b83fc9f50c3348ce13debe31eff302"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.441036 10699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:09.469343 10699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:09.472748 10699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:09.472932 10699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:09.473569 10699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:09.474466 10699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:09.474514 10699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:09.474555 10699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:09.474583 10699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:09.483597 10699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41527
I20260620 04:31:09.483629 10827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41527 every 8 connection(s)
I20260620 04:31:09.484529 10699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:31:09.486241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 10699
I20260620 04:31:09.486351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:31:09.486636 392 external_mini_cluster.cc:1428] Setting key 5265ecb98543597163aa157c9e83e66b
Jun 20 04:31:09 dist-test-slave-rzqm krb5kdc[10592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:31:09.498157 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.486454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47821 (local address 127.0.98.62:43573)
0620 04:31:09.486574 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:09.486578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:09.487628 (+ 1050us) server_negotiation.cc:407] Connection header received
0620 04:31:09.488989 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:09.488992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:09.489068 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:09.489149 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:09.490359 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.490885 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.492041 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.492273 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.492363 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.494776 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:09.494780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:09.494782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:31:09.494806 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:09.496112 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:09.496606 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:09.496610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:09.496612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:09.496685 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:09.497044 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:09.497047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:09.497049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:09.497232 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:09.497326 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:09.497855 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 04:31:09.497991 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:31:09.499114 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.486954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43573 (local address 127.0.98.1:47821)
0620 04:31:09.487464 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:09.487522 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:09.488763 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:09.489333 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:09.489341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:09.489699 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:09.490214 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.490228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.491004 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.491007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.491924 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.491934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.492414 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.492418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.492579 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.493169 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:09.493185 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:09.494605 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:09.496226 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:09.496233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:09.496250 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:09.496511 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:09.496772 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:09.496775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:09.496778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:09.496926 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:09.497340 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:09.497348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:09.497608 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 04:31:09.497807 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 04:31:09.498161 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":119,"threads_started":1}
I20260620 04:31:09.500523 10828 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43573
I20260620 04:31:09.500779 10828 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:09.501317 10828 heartbeater.cc:511] Master 127.0.98.62:43573 requested a full tablet report, sending...
I20260620 04:31:09.502885 10637 ts_manager.cc:194] Registered new tserver with Master: db23e4c03705476db27cea6dae964b26 (127.0.98.1:41527)
I20260620 04:31:09.504014 10637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:31:09.513768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43573
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:31:09.625952 10833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:09.626245 10833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:09.626289 10833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:09.630358 10833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:09.630427 10833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:09.630487 10833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:31:09.635958 10833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:09.637094 10833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:09.638175 10833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:09.648015 10838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:09.648094 10839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:09.648216 10833 server_base.cc:1034] running on GCE node
W20260620 04:31:09.648102 10841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:09.648751 10833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:09.649518 10833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:09.650699 10833 hybrid_clock.cc:648] HybridClock initialized: now 1781929869650680 us; error 41 us; skew 500 ppm
Jun 20 04:31:09 dist-test-slave-rzqm krb5kdc[10592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:31:09.653573 10833 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:31:09.654502 10833 webserver.cc:460] Webserver started at http://127.0.98.2:39627/ using document root <none> and password file <none>
I20260620 04:31:09.655031 10833 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:09.655115 10833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:09.655318 10833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:09.657364 10833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "86f599d2c6304ae0b62e2af3762f8dd3"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "c04794fa87357694afcae9315a242224"
server_key_iv: "aa0651d5a4fa2ffe3008c5b22326725d"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.657948 10833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "86f599d2c6304ae0b62e2af3762f8dd3"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "c04794fa87357694afcae9315a242224"
server_key_iv: "aa0651d5a4fa2ffe3008c5b22326725d"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.661654 10833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:31:09.664162 10848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:09.665666 10833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:31:09.665779 10833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "86f599d2c6304ae0b62e2af3762f8dd3"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "c04794fa87357694afcae9315a242224"
server_key_iv: "aa0651d5a4fa2ffe3008c5b22326725d"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.665897 10833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:09.685739 10833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:09.688959 10833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:09.689173 10833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:09.689777 10833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:09.690626 10833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:09.690673 10833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:09.690719 10833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:09.690747 10833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:09.699841 10833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40719
I20260620 04:31:09.699867 10961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40719 every 8 connection(s)
I20260620 04:31:09.700728 10833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 04:31:09 dist-test-slave-rzqm krb5kdc[10592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:31:09.710453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 10833
I20260620 04:31:09.710552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:31:09.710811 392 external_mini_cluster.cc:1428] Setting key ea6dbed0ad1f5cbe85e0c31b700e080e
I20260620 04:31:09.714726 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.702526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38433 (local address 127.0.98.62:43573)
0620 04:31:09.702668 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:09.702672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:09.703553 (+ 881us) server_negotiation.cc:407] Connection header received
0620 04:31:09.704831 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:09.704835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:09.704892 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:09.705052 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:09.706210 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.706747 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.707891 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.708122 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.708212 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.710791 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:09.710794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:09.710796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:31:09.710821 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:09.712508 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:09.713055 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:09.713058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:09.713060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:09.713153 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:09.713469 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:09.713472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:09.713474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:09.713692 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:09.713783 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:09.714396 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 04:31:09.714568 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:31:09.715587 10964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.702910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43573 (local address 127.0.98.2:38433)
0620 04:31:09.703364 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:09.703412 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:09.704619 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:09.705200 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:09.705207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:09.705586 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:09.706068 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.706083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.706869 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.706872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.707775 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.707785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.708264 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.708267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.708435 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.709077 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:09.709094 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:09.710607 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:09.712654 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:09.712660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:09.712674 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:09.712937 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:09.713245 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:09.713248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:09.713249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:09.713393 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:09.713811 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:09.713817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:09.714151 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 04:31:09.714336 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 04:31:09.714681 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
I20260620 04:31:09.716833 10962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43573
I20260620 04:31:09.717126 10962 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:09.717648 10962 heartbeater.cc:511] Master 127.0.98.62:43573 requested a full tablet report, sending...
I20260620 04:31:09.718747 10637 ts_manager.cc:194] Registered new tserver with Master: 86f599d2c6304ae0b62e2af3762f8dd3 (127.0.98.2:40719)
I20260620 04:31:09.719324 10637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38433
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:31:09.738150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43573
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:31:09.850499 10967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:09.850780 10967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:09.850826 10967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:09.854967 10967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:09.855031 10967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:09.855104 10967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:31:09.860450 10967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:09.861613 10967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:09.862634 10967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:09.872347 10975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:09.872347 10972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:09.872578 10967 server_base.cc:1034] running on GCE node
W20260620 04:31:09.872396 10973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:09.873225 10967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:09.873857 10967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:09.875037 10967 hybrid_clock.cc:648] HybridClock initialized: now 1781929869875017 us; error 33 us; skew 500 ppm
Jun 20 04:31:09 dist-test-slave-rzqm krb5kdc[10592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:31:09.877821 10967 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:31:09.878731 10967 webserver.cc:460] Webserver started at http://127.0.98.3:33571/ using document root <none> and password file <none>
I20260620 04:31:09.879277 10967 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:09.879343 10967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:09.879540 10967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:31:09.881688 10967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f04725bb96e4dca87304ea204afbaf2"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "69690279ee7bdcd0695f4c53cadf1e13"
server_key_iv: "582925cd17b6d0e4ace8df9142c5f5e3"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.882258 10967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f04725bb96e4dca87304ea204afbaf2"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "69690279ee7bdcd0695f4c53cadf1e13"
server_key_iv: "582925cd17b6d0e4ace8df9142c5f5e3"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.886380 10967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:31:09.889107 10982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:09.890594 10967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:31:09.890722 10967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "6f04725bb96e4dca87304ea204afbaf2"
format_stamp: "Formatted at 2026-06-20 04:31:09 on dist-test-slave-rzqm"
server_key: "69690279ee7bdcd0695f4c53cadf1e13"
server_key_iv: "582925cd17b6d0e4ace8df9142c5f5e3"
server_key_version: "encryptionkey@0"
I20260620 04:31:09.890838 10967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:09.918116 10967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:09.921778 10967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:09.921972 10967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:09.922650 10967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:09.923532 10967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:09.923581 10967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:09.923629 10967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:09.923657 10967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:09.931921 10967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42789
I20260620 04:31:09.931947 11095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42789 every 8 connection(s)
I20260620 04:31:09.932812 10967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:31:09.934674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 10967
I20260620 04:31:09.934801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:31:09.935089 392 external_mini_cluster.cc:1428] Setting key 43432853c451f6fa43756679e0f53439
Jun 20 04:31:09 dist-test-slave-rzqm krb5kdc[10592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:31:09.946736 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.934849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59637 (local address 127.0.98.62:43573)
0620 04:31:09.935012 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:09.935017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:09.935913 (+ 896us) server_negotiation.cc:407] Connection header received
0620 04:31:09.937560 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:09.937563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:09.937614 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:09.937689 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:09.938946 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.939464 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.940577 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.940789 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.940869 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.943347 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:09.943351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:09.943353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:31:09.943377 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:09.944699 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:09.945349 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:09.945352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:09.945353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:09.945435 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:09.945736 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:09.945738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:09.945739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:09.945901 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:09.945969 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:09.946503 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 04:31:09.946612 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:31:09.947554 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.935259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43573 (local address 127.0.98.3:59637)
0620 04:31:09.935758 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:09.935818 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:09.937347 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:09.937898 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:09.937909 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:09.938302 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:09.938816 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.938831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.939576 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.939578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.940463 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.940472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.940939 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.940943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.941204 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.941711 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:09.941726 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:09.943179 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:09.944860 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:09.944865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:09.944879 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:09.945252 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:09.945520 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:09.945523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:09.945524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:09.945662 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:09.946006 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:09.946011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:09.946253 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 04:31:09.946402 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 04:31:09.946667 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260620 04:31:09.948813 11096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43573
I20260620 04:31:09.949110 11096 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:09.949666 11096 heartbeater.cc:511] Master 127.0.98.62:43573 requested a full tablet report, sending...
I20260620 04:31:09.950659 10637 ts_manager.cc:194] Registered new tserver with Master: 6f04725bb96e4dca87304ea204afbaf2 (127.0.98.3:42789)
I20260620 04:31:09.951222 10637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59637
I20260620 04:31:09.961521 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:31:09.971014 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.963606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.0.98.62:43573)
0620 04:31:09.963792 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:09.963797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:09.963953 (+ 156us) server_negotiation.cc:407] Connection header received
0620 04:31:09.964082 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:09.964086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:09.964130 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:09.964193 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:09.964921 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.965501 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.966315 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.966543 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.966630 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.967935 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:09.967939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:09.967942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:31:09.967970 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:09.969415 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:09.969813 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:09.969816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:09.969818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:09.969900 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:09.970135 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:09.970138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:09.970140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:09.970392 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:09.970512 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:09.970730 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 04:31:09.970857 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 04:31:09.973860 10637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:31:09.976852 10637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:31:09.991884 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.985653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41527 (local address 127.0.0.1:39992)
0620 04:31:09.986302 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:09.986342 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:09.986481 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:09.987045 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:09.987051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:09.987077 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:09.987244 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.987252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.988945 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.988949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.990357 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.990368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.991457 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.991460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.991539 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.991559 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:31:09.991722 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 04:31:09.991791 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":53,"thread_start_us":114,"threads_started":1}
I20260620 04:31:09.992053 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.985823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.0.98.1:41527)
0620 04:31:09.986111 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:09.986128 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:09.986444 (+ 316us) server_negotiation.cc:407] Connection header received
0620 04:31:09.986637 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:09.986644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:09.986869 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:09.987422 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:09.987596 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.988804 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.990481 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.991264 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.991352 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.991476 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:09.991748 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 04:31:09.991832 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260620 04:31:09.992084 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.987127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.0.98.2:40719)
0620 04:31:09.987761 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:09.987771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:09.987788 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:31:09.987845 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:09.987848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:09.987969 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:09.988095 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:09.988303 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.989120 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.990526 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.991107 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.991214 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.991381 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:09.991714 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 04:31:09.991824 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":101,"threads_started":1}
I20260620 04:31:09.992477 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.986513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.0.98.3:42789)
0620 04:31:09.986992 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:09.987004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:09.987026 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:31:09.987099 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:09.987104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:09.987295 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:09.987448 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:09.987754 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.988823 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.990343 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.991191 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.991297 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.991443 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:09.992242 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 04:31:09.992350 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":176,"threads_started":1}
I20260620 04:31:09.991883 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.986745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40719 (local address 127.0.0.1:52564)
0620 04:31:09.987489 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:09.987501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:09.987578 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:09.988095 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:09.988099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:09.988114 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:09.988199 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.988206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.989282 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.989286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.990412 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.990419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.991220 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.991224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.991337 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.991375 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 04:31:09.991495 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 04:31:09.991710 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260620 04:31:09.992148 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:09.985720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42789 (local address 127.0.0.1:42056)
0620 04:31:09.986718 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:09.986730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:09.986806 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:09.987513 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:09.987516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:09.987533 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:09.987628 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.987634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.988975 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.988978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.990208 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:09.990217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:09.991507 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:09.991511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:09.991585 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:09.991905 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 04:31:09.991972 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:31:09.992041 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":33,"thread_start_us":86,"threads_started":1}
I20260620 04:31:09.995584 10762 tablet_service.cc:1467] Processing CreateTablet for tablet ce3a6762e83146d2a7b400f74d8a8ede (DEFAULT_TABLE table=test-table [id=9bcc87c4a1824f7d82e1957d18c28fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:31:09.995577 11030 tablet_service.cc:1467] Processing CreateTablet for tablet ce3a6762e83146d2a7b400f74d8a8ede (DEFAULT_TABLE table=test-table [id=9bcc87c4a1824f7d82e1957d18c28fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:31:09.996745 10762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3a6762e83146d2a7b400f74d8a8ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:31:09.996752 11030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3a6762e83146d2a7b400f74d8a8ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:31:09.996538 10896 tablet_service.cc:1467] Processing CreateTablet for tablet ce3a6762e83146d2a7b400f74d8a8ede (DEFAULT_TABLE table=test-table [id=9bcc87c4a1824f7d82e1957d18c28fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:31:09.997455 10896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3a6762e83146d2a7b400f74d8a8ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:31:10.003973 11114 tablet_bootstrap.cc:492] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26: Bootstrap starting.
I20260620 04:31:10.005352 11113 tablet_bootstrap.cc:492] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: Bootstrap starting.
I20260620 04:31:10.006270 11114 tablet_bootstrap.cc:654] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26: Neither blocks nor log segments found. Creating new log.
I20260620 04:31:10.006986 11114 log.cc:824] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26: Log is configured to *not* fsync() on all Append() calls
I20260620 04:31:10.007483 11113 tablet_bootstrap.cc:654] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: Neither blocks nor log segments found. Creating new log.
I20260620 04:31:10.007673 11115 tablet_bootstrap.cc:492] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: Bootstrap starting.
I20260620 04:31:10.008329 11113 log.cc:824] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: Log is configured to *not* fsync() on all Append() calls
I20260620 04:31:10.009202 11114 tablet_bootstrap.cc:492] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26: No bootstrap required, opened a new log
I20260620 04:31:10.009451 11114 ts_tablet_manager.cc:1397] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:31:10.009905 11115 tablet_bootstrap.cc:654] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: Neither blocks nor log segments found. Creating new log.
I20260620 04:31:10.010524 11113 tablet_bootstrap.cc:492] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: No bootstrap required, opened a new log
I20260620 04:31:10.010713 11113 ts_tablet_manager.cc:1397] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:31:10.011125 11115 log.cc:824] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: Log is configured to *not* fsync() on all Append() calls
I20260620 04:31:10.013077 11114 raft_consensus.cc:348] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.013280 11115 tablet_bootstrap.cc:492] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: No bootstrap required, opened a new log
I20260620 04:31:10.013303 11114 raft_consensus.cc:374] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:31:10.013389 11114 raft_consensus.cc:729] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db23e4c03705476db27cea6dae964b26, State: Initialized, Role: FOLLOWER
I20260620 04:31:10.013454 11115 ts_tablet_manager.cc:1397] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:31:10.013860 11114 consensus_queue.cc:260] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.014503 10828 heartbeater.cc:503] Master 127.0.98.62:43573 was elected leader, sending a full tablet report...
I20260620 04:31:10.014379 11113 raft_consensus.cc:348] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.014648 11113 raft_consensus.cc:374] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:31:10.014694 11113 raft_consensus.cc:729] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f599d2c6304ae0b62e2af3762f8dd3, State: Initialized, Role: FOLLOWER
I20260620 04:31:10.014873 11114 ts_tablet_manager.cc:1428] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:31:10.015144 11113 consensus_queue.cc:260] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.016096 11113 ts_tablet_manager.cc:1428] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:31:10.016469 10962 heartbeater.cc:503] Master 127.0.98.62:43573 was elected leader, sending a full tablet report...
I20260620 04:31:10.017187 11115 raft_consensus.cc:348] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.017491 11115 raft_consensus.cc:374] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:31:10.017561 11115 raft_consensus.cc:729] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f04725bb96e4dca87304ea204afbaf2, State: Initialized, Role: FOLLOWER
I20260620 04:31:10.018006 11115 consensus_queue.cc:260] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.018774 11096 heartbeater.cc:503] Master 127.0.98.62:43573 was elected leader, sending a full tablet report...
I20260620 04:31:10.019006 11115 ts_tablet_manager.cc:1428] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 04:31:10.183776 11097 tablet.cc:2369] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:31:10.201897 10963 tablet.cc:2369] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:31:10.235903 10829 tablet.cc:2369] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:31:10.337180 11121 raft_consensus.cc:482] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:31:10.337399 11121 raft_consensus.cc:504] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.338640 11121 leader_election.cc:288] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db23e4c03705476db27cea6dae964b26 (127.0.98.1:41527), 86f599d2c6304ae0b62e2af3762f8dd3 (127.0.98.2:40719)
I20260620 04:31:10.339476 11120 raft_consensus.cc:482] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:31:10.339721 11120 raft_consensus.cc:504] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.341468 11120 leader_election.cc:288] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db23e4c03705476db27cea6dae964b26 (127.0.98.1:41527), 6f04725bb96e4dca87304ea204afbaf2 (127.0.98.3:42789)
I20260620 04:31:10.343180 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.338940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41527 (local address 127.0.98.3:34659)
0620 04:31:10.339120 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:10.339136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:10.339233 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:10.339513 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:10.339516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:10.339535 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:10.339648 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:10.339654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.340470 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.340473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:10.341683 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:10.341694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.342630 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.342633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:10.342755 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:10.342892 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 04:31:10.342962 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 04:31:10.343040 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":108}
I20260620 04:31:10.343345 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.339303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44023 (local address 127.0.98.2:40719)
0620 04:31:10.339419 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:10.339424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:10.339550 (+ 126us) server_negotiation.cc:407] Connection header received
0620 04:31:10.339736 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:10.339740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:10.339787 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:10.339889 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:10.340264 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.340791 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.342133 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.342702 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.342765 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:10.342825 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:10.343156 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 04:31:10.343229 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:31:10.343669 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.339013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40719 (local address 127.0.98.3:44023)
0620 04:31:10.339470 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:10.339484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:10.339623 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:10.340022 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:10.340025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:10.340041 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:10.340147 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:10.340152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.340893 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.340897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:10.341996 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:10.342006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.342925 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.342928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:10.343031 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:10.343052 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:31:10.343466 (+ 414us) client_negotiation.cc:240] Negotiation successful
0620 04:31:10.343537 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":1,"thread_start_us":139,"threads_started":1}
I20260620 04:31:10.343930 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.339038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34659 (local address 127.0.98.1:41527)
0620 04:31:10.339168 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:10.339172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:10.339188 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:31:10.339362 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:10.339366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:10.339415 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:10.339480 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:10.339744 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.340333 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.341815 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.342411 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.342547 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:10.342856 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:10.343719 (+ 863us) server_negotiation.cc:299] Negotiation successful
0620 04:31:10.343796 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":205,"server-negotiator.queue_time_us":54}
I20260620 04:31:10.344545 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3a6762e83146d2a7b400f74d8a8ede" candidate_uuid: "6f04725bb96e4dca87304ea204afbaf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db23e4c03705476db27cea6dae964b26" is_pre_election: true
I20260620 04:31:10.344738 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3a6762e83146d2a7b400f74d8a8ede" candidate_uuid: "6f04725bb96e4dca87304ea204afbaf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" is_pre_election: true
I20260620 04:31:10.344872 10782 raft_consensus.cc:2454] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f04725bb96e4dca87304ea204afbaf2 in term 0.
I20260620 04:31:10.344949 10916 raft_consensus.cc:2454] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f04725bb96e4dca87304ea204afbaf2 in term 0.
I20260620 04:31:10.345499 10986 leader_election.cc:302] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f04725bb96e4dca87304ea204afbaf2, 86f599d2c6304ae0b62e2af3762f8dd3; no voters:
I20260620 04:31:10.346092 11121 raft_consensus.cc:2781] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:31:10.346328 11121 raft_consensus.cc:482] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:31:10.346562 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.341159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41527 (local address 127.0.98.2:50357)
0620 04:31:10.341663 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:10.341676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:10.341763 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:10.342219 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:10.342222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:10.342239 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:10.342372 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:10.342378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.343403 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.343407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:10.345273 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:10.345283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.346111 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.346114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:10.346214 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:10.346237 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:31:10.346343 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 04:31:10.346417 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260620 04:31:10.346572 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.341491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50357 (local address 127.0.98.1:41527)
0620 04:31:10.342007 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:10.342014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:10.342026 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:31:10.342063 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:10.342065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:10.342105 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:10.342468 (+ 363us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:10.342571 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.343265 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.345397 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.345984 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.346087 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:10.346152 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:10.346359 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 04:31:10.346438 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":441,"thread_start_us":99,"threads_started":1}
I20260620 04:31:10.346582 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.341771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58707 (local address 127.0.98.3:42789)
0620 04:31:10.341868 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:10.341872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:10.342104 (+ 232us) server_negotiation.cc:407] Connection header received
0620 04:31:10.342254 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:10.342257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:10.342297 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:10.342349 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:10.342712 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.343424 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.345480 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.345979 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.346042 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:10.346097 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:10.346383 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 04:31:10.346447 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":39}
I20260620 04:31:10.346710 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.341203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42789 (local address 127.0.98.2:58707)
0620 04:31:10.342031 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:10.342044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:10.342160 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:10.342469 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:10.342472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:10.342487 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:10.342596 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:10.342603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.343586 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.343589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:10.345354 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:10.345366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.346172 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.346175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:10.346258 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:10.346282 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:31:10.346502 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 04:31:10.346559 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":67,"threads_started":1}
I20260620 04:31:10.346480 11121 raft_consensus.cc:3037] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:31:10.347182 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3a6762e83146d2a7b400f74d8a8ede" candidate_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db23e4c03705476db27cea6dae964b26" is_pre_election: true
I20260620 04:31:10.347308 10782 raft_consensus.cc:2454] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86f599d2c6304ae0b62e2af3762f8dd3 in term 0.
I20260620 04:31:10.347337 11050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3a6762e83146d2a7b400f74d8a8ede" candidate_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f04725bb96e4dca87304ea204afbaf2" is_pre_election: true
I20260620 04:31:10.347635 10849 leader_election.cc:302] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86f599d2c6304ae0b62e2af3762f8dd3, db23e4c03705476db27cea6dae964b26; no voters:
I20260620 04:31:10.347918 11120 raft_consensus.cc:2781] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:31:10.348016 11120 raft_consensus.cc:482] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:31:10.348069 11120 raft_consensus.cc:3037] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:31:10.349211 11120 raft_consensus.cc:504] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.349561 11120 leader_election.cc:288] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [CANDIDATE]: Term 1 election: Requested vote from peers db23e4c03705476db27cea6dae964b26 (127.0.98.1:41527), 6f04725bb96e4dca87304ea204afbaf2 (127.0.98.3:42789)
I20260620 04:31:10.349967 10782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3a6762e83146d2a7b400f74d8a8ede" candidate_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db23e4c03705476db27cea6dae964b26"
I20260620 04:31:10.349961 11049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3a6762e83146d2a7b400f74d8a8ede" candidate_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f04725bb96e4dca87304ea204afbaf2"
I20260620 04:31:10.349970 11121 raft_consensus.cc:504] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.350095 10782 raft_consensus.cc:3037] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:31:10.350260 11050 raft_consensus.cc:2379] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86f599d2c6304ae0b62e2af3762f8dd3 in current term 1: Already voted for candidate 6f04725bb96e4dca87304ea204afbaf2 in this term.
I20260620 04:31:10.350304 11121 leader_election.cc:288] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [CANDIDATE]: Term 1 election: Requested vote from peers db23e4c03705476db27cea6dae964b26 (127.0.98.1:41527), 86f599d2c6304ae0b62e2af3762f8dd3 (127.0.98.2:40719)
I20260620 04:31:10.351222 10781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3a6762e83146d2a7b400f74d8a8ede" candidate_uuid: "6f04725bb96e4dca87304ea204afbaf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db23e4c03705476db27cea6dae964b26"
I20260620 04:31:10.351392 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3a6762e83146d2a7b400f74d8a8ede" candidate_uuid: "6f04725bb96e4dca87304ea204afbaf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86f599d2c6304ae0b62e2af3762f8dd3"
I20260620 04:31:10.351445 10782 raft_consensus.cc:2454] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86f599d2c6304ae0b62e2af3762f8dd3 in term 1.
I20260620 04:31:10.351547 10916 raft_consensus.cc:2379] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f04725bb96e4dca87304ea204afbaf2 in current term 1: Already voted for candidate 86f599d2c6304ae0b62e2af3762f8dd3 in this term.
I20260620 04:31:10.351737 10849 leader_election.cc:302] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 86f599d2c6304ae0b62e2af3762f8dd3, db23e4c03705476db27cea6dae964b26; no voters: 6f04725bb96e4dca87304ea204afbaf2
I20260620 04:31:10.351903 11120 raft_consensus.cc:2781] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:31:10.351903 10986 leader_election.cc:302] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f04725bb96e4dca87304ea204afbaf2; no voters: 86f599d2c6304ae0b62e2af3762f8dd3, db23e4c03705476db27cea6dae964b26
I20260620 04:31:10.352048 11121 raft_consensus.cc:2726] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 04:31:10.352123 11120 raft_consensus.cc:686] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 86f599d2c6304ae0b62e2af3762f8dd3, State: Running, Role: LEADER
I20260620 04:31:10.352432 11120 consensus_queue.cc:237] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } }
I20260620 04:31:10.355635 10636 catalog_manager.cc:5495] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86f599d2c6304ae0b62e2af3762f8dd3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86f599d2c6304ae0b62e2af3762f8dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40719 } health_report { overall_health: HEALTHY } } }
I20260620 04:31:10.445484 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.441725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.0.98.2:40719)
0620 04:31:10.441852 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:10.441855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:10.441869 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:31:10.441948 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:10.441950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:10.442002 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:10.442094 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:31:10.442356 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.442908 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.443830 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:10.444059 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:10.444261 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:10.444545 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:31:10.444992 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:31:10.445168 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:10.445271 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 04:31:10.445363 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:31:10.451814 10781 raft_consensus.cc:1261] T ce3a6762e83146d2a7b400f74d8a8ede P db23e4c03705476db27cea6dae964b26 [term 1 FOLLOWER]: Refusing update from remote peer 86f599d2c6304ae0b62e2af3762f8dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:31:10.451840 11050 raft_consensus.cc:1261] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 1 FOLLOWER]: Refusing update from remote peer 86f599d2c6304ae0b62e2af3762f8dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:31:10.452562 11120 consensus_queue.cc:1035] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f04725bb96e4dca87304ea204afbaf2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:31:10.452816 11126 consensus_queue.cc:1035] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db23e4c03705476db27cea6dae964b26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:31:10.460680 11129 mvcc.cc:205] Tried to move back new op lower bound from 7298784749363593216 to 7298784748966494208. Current Snapshot: MvccSnapshot[applied={T|T < 7298784749363593216}]
I20260620 04:31:10.461194 11131 mvcc.cc:205] Tried to move back new op lower bound from 7298784749363593216 to 7298784748966494208. Current Snapshot: MvccSnapshot[applied={T|T < 7298784749363593216}]
I20260620 04:31:10.463624 11130 mvcc.cc:205] Tried to move back new op lower bound from 7298784749363593216 to 7298784748966494208. Current Snapshot: MvccSnapshot[applied={T|T < 7298784749363593216}]
I20260620 04:31:10.469815 10637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:31:10.469993 10637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:31:10.472499 10637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d26ca13196a4d348b3033468621bba3: Sending DeleteTablet for 3 replicas of tablet ce3a6762e83146d2a7b400f74d8a8ede
I20260620 04:31:10.473376 10762 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3a6762e83146d2a7b400f74d8a8ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:31:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39992
I20260620 04:31:10.473363 11030 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3a6762e83146d2a7b400f74d8a8ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:31:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42056
I20260620 04:31:10.473428 10896 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3a6762e83146d2a7b400f74d8a8ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:31:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52564
I20260620 04:31:10.474321 11138 tablet_replica.cc:330] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: stopping tablet replica
I20260620 04:31:10.474447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 10699
I20260620 04:31:10.474660 11138 raft_consensus.cc:2229] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:31:10.474987 11138 raft_consensus.cc:2258] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:31:10.475330 11139 tablet_replica.cc:330] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: stopping tablet replica
I20260620 04:31:10.475628 11139 raft_consensus.cc:2229] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:31:10.476025 11139 raft_consensus.cc:2258] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:31:10.476823 11138 ts_tablet_manager.cc:1905] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:31:10.479192 11138 ts_tablet_manager.cc:1918] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:31:10.479292 11138 log.cc:1196] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/ce3a6762e83146d2a7b400f74d8a8ede
I20260620 04:31:10.479588 11138 ts_tablet_manager.cc:1939] T ce3a6762e83146d2a7b400f74d8a8ede P 6f04725bb96e4dca87304ea204afbaf2: Deleting consensus metadata
I20260620 04:31:10.480765 10622 catalog_manager.cc:4841] TS 6f04725bb96e4dca87304ea204afbaf2 (127.0.98.3:42789): tablet ce3a6762e83146d2a7b400f74d8a8ede (table test-table [id=9bcc87c4a1824f7d82e1957d18c28fba]) successfully deleted
I20260620 04:31:10.481916 11139 ts_tablet_manager.cc:1905] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:31:10.483419 10621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41527 (error 108)
I20260620 04:31:10.484500 11139 ts_tablet_manager.cc:1918] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:31:10.484642 11139 log.cc:1196] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/ce3a6762e83146d2a7b400f74d8a8ede
I20260620 04:31:10.484932 11139 ts_tablet_manager.cc:1939] T ce3a6762e83146d2a7b400f74d8a8ede P 86f599d2c6304ae0b62e2af3762f8dd3: Deleting consensus metadata
I20260620 04:31:10.485045 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.484414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41527 (local address 127.0.0.1:39998)
0620 04:31:10.484850 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:10.484926 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321}
W20260620 04:31:10.485227 10621 catalog_manager.cc:4584] TS db23e4c03705476db27cea6dae964b26 (127.0.98.1:41527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce3a6762e83146d2a7b400f74d8a8ede: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41527: connect: Connection refused (error 111)
I20260620 04:31:10.485432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 10833
I20260620 04:31:10.491418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 10967
I20260620 04:31:10.491626 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:10.491361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40719 (local address 127.0.0.1:52592)
0620 04:31:10.491492 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:10.491545 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 04:31:10.496503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 10605
2026-06-20T04:31:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2588 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:31:10 dist-test-slave-rzqm krb5kdc[11144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:31:10 dist-test-slave-rzqm krb5kdc[11144](info): set up 2 sockets
Jun 20 04:31:10 dist-test-slave-rzqm krb5kdc[11144](info): commencing operation
krb5kdc: starting...
W20260620 04:31:11.457788 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:31:11 dist-test-slave-rzqm krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:31:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 04:31:11.624449 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYF3Tc0/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/postgres -l logfile start
2026-06-20 04:31:14.175 UTC [11174] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 04:31:14.175 UTC [11174] LOG: listening on IPv4 address "127.0.98.20", port 39253
2026-06-20 04:31:14.179 UTC [11174] LOG: listening on Unix socket "/tmp/.s.PGSQL.39253"
2026-06-20 04:31:14.206 UTC [11177] LOG: database system was shut down at 2026-06-20 04:31:12 UTC
2026-06-20 04:31:14.210 UTC [11174] LOG: database system is ready to accept connections
I20260620 04:31:15.089526 392 mini_postgres.cc:96] Postgres bound to 39253
2026-06-20 04:31:15.095 UTC [11185] FATAL: database "slave" does not exist
127.0.98.20:39253 - accepting connections
I20260620 04:31:15.096055 392 mini_ranger.cc:161] Starting Ranger...
I20260620 04:31:15.110530 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 04:31:15.637907 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 04:31:15.637997 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-admin
2026-06-20 04:31:15,907 [I] DB FLAVOR :POSTGRES
2026-06-20 04:31:15,908 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:31:15,908 [I] Checking connection..
2026-06-20 04:31:15,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 04:31:16,160 [I] Checking connection passed.
2026-06-20 04:31:16,161 [I] --------- Verifying version history table ---------
2026-06-20 04:31:16,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:31:16,429 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:31:16,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:31:16,691 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:31:16,691 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 04:31:16,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 04:31:16.938 UTC [11249] WARNING: there is no transaction in progress
2026-06-20 04:31:16,944 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 04:31:16,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:31:17,191 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 04:31:17,192 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:31:17,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 04:31:17,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 04:31:17,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rzqm.c.gcp-upstream.internal','N') ;"
2026-06-20 04:31:17,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:31:18,162 [I] Table x_portal_user does not exist in database ranger
2026-06-20 04:31:18,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:31:18,426 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 04:31:18,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:31:18,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:31:18,912 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 04:31:18,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 04:31:19.894 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:19.910 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:19.928 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:19.945 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:19.962 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:19.981 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20.048 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20.056 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20.071 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20.081 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20.091 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20.106 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20.116 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20.123 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20.131 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20.139 UTC [11357] WARNING: there is no transaction in progress
2026-06-20 04:31:20,886 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 04:31:20,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 04:31:21,127 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 04:31:21,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:31:21,380 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 04:31:21,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:31:21,639 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 04:31:21,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:31:21,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:31:22,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rzqm.c.gcp-upstream.internal';"
2026-06-20 04:31:22,422 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 04:31:22,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 04:31:22,684 [I] DB_PATCHES have already been applied
I20260620 04:31:22.690531 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYF3Tc0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 04:31:22.690600 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 04:31:22.693913 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:42713
Jun 20, 2026 4:31:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:31:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:31:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 4:31:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 4:31:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 4:31:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 4:31:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 04:31:23.682613 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.587s user 0.003s sys 0.006s
Jun 20, 2026 4:31:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 4:31:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 4:31:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 4:31:44 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 4:31:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 04:31:45.710510 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 04:31:45 dist-test-slave-rzqm krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929905, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 04:31:45.798106 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 04:31:45.807572 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 04:31:46.713696 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 04:31:46.713778 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-kms
I20260620 04:31:46.713835 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:39253
2026-06-20 04:31:46,888 [I] DB FLAVOR :POSTGRES
2026-06-20 04:31:46,888 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:31:46,888 [I] Checking connection
2026-06-20 04:31:46,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 04:31:47,128 [I] connection success
2026-06-20 04:31:47,128 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 04:31:47,128 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 04:31:47,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 04:31:47,370 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 04:31:47,370 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:31:47,371 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 04:31:47,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 04:31:47,641 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 04:31:48.141494 392 mini_ranger_kms.cc:310] Created kms service
I20260620 04:31:48.308902 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 04:31:48.353555 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 04:31:48 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929905, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:31:48.761329 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 04:31:48.761476 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 04:31:48.761497 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 04:31:48.765210 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11537
I20260620 04:31:48.765374 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:42995
Jun 20, 2026 4:31:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:31:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:31:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 4:31:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 4:31:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 4:31:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 04:31:50.809284 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.011s user 0.017s sys 0.142s
I20260620 04:31:50.809415 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 04:31:50.809527 392 mini_ranger_kms.cc:401] 127.0.98.20:42995/kms/v1/keys
Jun 20 04:31:50 dist-test-slave-rzqm krb5kdc[11144](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781929910, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 04:31:51 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929910, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:31:51 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929910, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:31:54 dist-test-slave-rzqm krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:31:54.070274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35447
--ranger_config_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42995/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:31:54.204123 11588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:54.204406 11588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:54.204445 11588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:54.208600 11588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:31:54.208673 11588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:54.208689 11588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:54.208709 11588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:31:54.208724 11588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:31:54.214475 11588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:42995/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35447
--ranger_config_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:54.215569 11588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:54.216677 11588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:54.226931 11593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:54.226950 11594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:54.226974 11596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:54.227241 11588 server_base.cc:1034] running on GCE node
I20260620 04:31:54.228009 11588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:54.229604 11588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:54.230793 11588 hybrid_clock.cc:648] HybridClock initialized: now 1781929914230772 us; error 37 us; skew 500 ppm
Jun 20 04:31:54 dist-test-slave-rzqm krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:31:54.240275 11588 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:31:54.241621 11588 webserver.cc:460] Webserver started at http://127.0.98.62:45485/ using document root <none> and password file <none>
I20260620 04:31:54.242354 11588 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:54.242439 11588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:54.242758 11588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:31:54 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:31:54.290652 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "a512aef8f17d4150b24f85649a522016"
format_stamp: "Formatted at 2026-06-20 04:31:54 on dist-test-slave-rzqm"
server_key: "18daa2e1a23db287e8a43209c575ef9e"
server_key_iv: "d84c63fe44dbaac5732513acd3231545"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:54.291368 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "a512aef8f17d4150b24f85649a522016"
format_stamp: "Formatted at 2026-06-20 04:31:54 on dist-test-slave-rzqm"
server_key: "18daa2e1a23db287e8a43209c575ef9e"
server_key_iv: "d84c63fe44dbaac5732513acd3231545"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:54.295608 11588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 04:31:54.313405 11603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:54.314746 11588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 04:31:54.314872 11588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "a512aef8f17d4150b24f85649a522016"
format_stamp: "Formatted at 2026-06-20 04:31:54 on dist-test-slave-rzqm"
server_key: "18daa2e1a23db287e8a43209c575ef9e"
server_key_iv: "d84c63fe44dbaac5732513acd3231545"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:54.315001 11588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:54.335383 11588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:54.338460 11588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:54.338647 11588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:54.346558 11588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35447
I20260620 04:31:54.346596 11655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35447 every 8 connection(s)
I20260620 04:31:54.347543 11588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:31:54.348820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 11588
I20260620 04:31:54.348930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:31:54.349247 392 external_mini_cluster.cc:1428] Setting key 32f088cb881798adc28e1823ef5fc5b4
I20260620 04:31:54.350661 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:31:54.357616 11656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016: Bootstrap starting.
Jun 20 04:31:54 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:31:54.360736 11656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016: Neither blocks nor log segments found. Creating new log.
I20260620 04:31:54.361963 11656 log.cc:824] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016: Log is configured to *not* fsync() on all Append() calls
I20260620 04:31:54.364084 11656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016: No bootstrap required, opened a new log
I20260620 04:31:54.365968 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:54.350724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.0.98.62:35447)
0620 04:31:54.351192 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:54.351211 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:54.351252 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:31:54.352077 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:54.352101 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:54.352545 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:54.352932 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:54.353857 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:54.355348 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:54.356593 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:54.357341 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:54.357496 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:54.360555 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:54.360563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:54.360583 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:31:54.360620 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:54.362575 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:54.363093 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:54.363099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:54.363106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:54.363240 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:54.363523 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:54.363526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:54.363528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:54.364067 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:54.364254 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:54.364652 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 04:31:54.364940 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":148,"threads_started":1}
I20260620 04:31:54.367480 11656 raft_consensus.cc:348] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a512aef8f17d4150b24f85649a522016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } }
I20260620 04:31:54.367774 11656 raft_consensus.cc:374] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:31:54.367832 11656 raft_consensus.cc:729] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a512aef8f17d4150b24f85649a522016, State: Initialized, Role: FOLLOWER
I20260620 04:31:54.368326 11656 consensus_queue.cc:260] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a512aef8f17d4150b24f85649a522016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } }
I20260620 04:31:54.368491 11656 raft_consensus.cc:388] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:31:54.368549 11656 raft_consensus.cc:482] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:31:54.368633 11656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:31:54.369671 11656 raft_consensus.cc:504] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a512aef8f17d4150b24f85649a522016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } }
I20260620 04:31:54.370014 11656 leader_election.cc:302] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a512aef8f17d4150b24f85649a522016; no voters:
I20260620 04:31:54.370342 11656 leader_election.cc:288] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:31:54.370424 11661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:31:54.370596 11661 raft_consensus.cc:686] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [term 1 LEADER]: Becoming Leader. State: Replica: a512aef8f17d4150b24f85649a522016, State: Running, Role: LEADER
I20260620 04:31:54.371680 11656 sys_catalog.cc:564] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:31:54.371232 11661 consensus_queue.cc:237] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a512aef8f17d4150b24f85649a522016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } }
I20260620 04:31:54.373345 11663 sys_catalog.cc:455] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a512aef8f17d4150b24f85649a522016. Latest consensus state: current_term: 1 leader_uuid: "a512aef8f17d4150b24f85649a522016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a512aef8f17d4150b24f85649a522016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } } }
I20260620 04:31:54.373481 11663 sys_catalog.cc:458] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [sys.catalog]: This master's current role is: LEADER
I20260620 04:31:54.373760 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a512aef8f17d4150b24f85649a522016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a512aef8f17d4150b24f85649a522016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35447 } } }
I20260620 04:31:54.373844 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016 [sys.catalog]: This master's current role is: LEADER
I20260620 04:31:54.373908 11656 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 04:31:54.374112 11675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:31:54.382026 11675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:31:54.389981 11675 catalog_manager.cc:1347] Generated new cluster ID: ecfc98da54604e6a98677f63db5127cb
I20260620 04:31:54.390066 11675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:31:54.407215 11675 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:31:54.408603 11675 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:31:54.420539 11675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016: Generated new TSK 0
I20260620 04:31:54.421648 11675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:31:55.183787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42995/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35447
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:31:55.351154 11701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:55.351545 11701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:55.351608 11701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:55.357857 11701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:55.358013 11701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:55.358166 11701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:31:55.366542 11701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:42995/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:55.368098 11701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:55.369678 11701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:55.382966 11709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:55.383315 11711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:55.383359 11701 server_base.cc:1034] running on GCE node
W20260620 04:31:55.383653 11708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:55.384140 11701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:55.385035 11701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:55.386245 11701 hybrid_clock.cc:648] HybridClock initialized: now 1781929915386217 us; error 42 us; skew 500 ppm
Jun 20 04:31:55 dist-test-slave-rzqm krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:31:55.389919 11701 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:31:55.391148 11701 webserver.cc:460] Webserver started at http://127.0.98.1:39213/ using document root <none> and password file <none>
I20260620 04:31:55.391920 11701 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:55.392016 11701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:55.392279 11701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:31:55 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:31:55.413801 11701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a70f73d631047bb8995a1105621e7ff"
format_stamp: "Formatted at 2026-06-20 04:31:55 on dist-test-slave-rzqm"
server_key: "1d84e9ed09005ae05970f99c8adb012c"
server_key_iv: "ecb1ce7b3e319995f0b2746dd58e3041"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:55.414817 11701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a70f73d631047bb8995a1105621e7ff"
format_stamp: "Formatted at 2026-06-20 04:31:55 on dist-test-slave-rzqm"
server_key: "1d84e9ed09005ae05970f99c8adb012c"
server_key_iv: "ecb1ce7b3e319995f0b2746dd58e3041"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:55.428799 11701 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
I20260620 04:31:55.447000 11720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:55.448923 11701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 04:31:55.449157 11701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "9a70f73d631047bb8995a1105621e7ff"
format_stamp: "Formatted at 2026-06-20 04:31:55 on dist-test-slave-rzqm"
server_key: "1d84e9ed09005ae05970f99c8adb012c"
server_key_iv: "ecb1ce7b3e319995f0b2746dd58e3041"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:55.449331 11701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:55.474684 11701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:55.478566 11701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:55.478832 11701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:55.479676 11701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:55.480822 11701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:55.480937 11701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:55.481108 11701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:55.481196 11701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:55.492657 11701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41849
I20260620 04:31:55.494050 11701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:31:55.494159 11833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41849 every 8 connection(s)
I20260620 04:31:55.497144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 11701
I20260620 04:31:55.497262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:31:55.497556 392 external_mini_cluster.cc:1428] Setting key 37aec3c7232a70ca735ad3b6a0f12b06
Jun 20 04:31:55 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:31:55.515008 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:55.496705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49325 (local address 127.0.98.62:35447)
0620 04:31:55.498257 (+ 1552us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:55.498262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:55.499881 (+ 1619us) server_negotiation.cc:407] Connection header received
0620 04:31:55.499927 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:55.499930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:55.499982 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:55.500057 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:55.502102 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:55.502953 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:55.504782 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:55.505173 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:55.505261 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:55.508361 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:55.508366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:55.508369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:31:55.508399 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:55.510391 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:55.511088 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:55.511092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:55.511094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:55.512721 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:55.513426 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:55.513430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:55.513432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:55.513789 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:55.513872 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:55.514653 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 04:31:55.514808 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":82,"threads_started":1}
I20260620 04:31:55.517170 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:55.497203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35447 (local address 127.0.98.1:49325)
0620 04:31:55.498532 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:55.498587 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:55.499582 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:55.500377 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:55.500386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:55.501225 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:55.501918 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:55.501938 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:55.503136 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:55.503140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:55.504621 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:55.504633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:55.505497 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:55.505501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:55.505720 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:55.506373 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:55.506394 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:55.508136 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:55.510532 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:55.510540 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:55.510558 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:55.510965 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:55.512863 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:55.512866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:55.512868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:55.513293 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:55.513953 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:55.513961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:55.514336 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 04:31:55.515551 (+ 1215us) client_negotiation.cc:240] Negotiation successful
0620 04:31:55.515958 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":141,"threads_started":1}
I20260620 04:31:55.519176 11834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35447
I20260620 04:31:55.519443 11834 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:55.520079 11834 heartbeater.cc:511] Master 127.0.98.62:35447 requested a full tablet report, sending...
I20260620 04:31:55.521955 11620 ts_manager.cc:194] Registered new tserver with Master: 9a70f73d631047bb8995a1105621e7ff (127.0.98.1:41849)
I20260620 04:31:55.523208 11620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:31:55.530342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42995/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35447
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:31:55 dist-test-slave-rzqm krb5kdc[11144](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781929915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 04:31:55.660167 11840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:55.660462 11840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:55.660524 11840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:55.664832 11840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:55.664914 11840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:55.665019 11840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:31:55.670727 11840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:42995/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:55.671947 11840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:55.673120 11840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:55.682935 11847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:55.683102 11840 server_base.cc:1034] running on GCE node
W20260620 04:31:55.682947 11846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:55.683291 11849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:55.684474 11840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:55.685266 11840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:55.686481 11840 hybrid_clock.cc:648] HybridClock initialized: now 1781929915686419 us; error 89 us; skew 500 ppm
Jun 20 04:31:55 dist-test-slave-rzqm krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:31:55.690099 11840 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:31:55.690992 11840 webserver.cc:460] Webserver started at http://127.0.98.2:37047/ using document root <none> and password file <none>
I20260620 04:31:55.691550 11840 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:55.691615 11840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:55.691825 11840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:31:55 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:31:55.712373 11840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "47ddafa0e3e1465c9cc65237a61c5aaf"
format_stamp: "Formatted at 2026-06-20 04:31:55 on dist-test-slave-rzqm"
server_key: "5180c61f82ff545f1ab6909990cf25ee"
server_key_iv: "3d3620634bc439756af6c67602cf27bb"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:55.713369 11840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "47ddafa0e3e1465c9cc65237a61c5aaf"
format_stamp: "Formatted at 2026-06-20 04:31:55 on dist-test-slave-rzqm"
server_key: "5180c61f82ff545f1ab6909990cf25ee"
server_key_iv: "3d3620634bc439756af6c67602cf27bb"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:55.721832 11840 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260620 04:31:55.747115 11856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:55.749063 11840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 04:31:55.749213 11840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "47ddafa0e3e1465c9cc65237a61c5aaf"
format_stamp: "Formatted at 2026-06-20 04:31:55 on dist-test-slave-rzqm"
server_key: "5180c61f82ff545f1ab6909990cf25ee"
server_key_iv: "3d3620634bc439756af6c67602cf27bb"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:55.749354 11840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:55.777038 11840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:55.781150 11840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:55.781435 11840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:55.782298 11840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:55.783478 11840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:55.783592 11840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:55.783696 11840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:55.783762 11840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:55.797781 11840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40219
I20260620 04:31:55.797783 11969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40219 every 8 connection(s)
I20260620 04:31:55.799063 11840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:31:55.807446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 11840
I20260620 04:31:55.807606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:31:55.807946 392 external_mini_cluster.cc:1428] Setting key 7baaec35a8d57e75309cbab3bae50fc4
Jun 20 04:31:55 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:31:55.820863 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:55.801283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33623 (local address 127.0.98.62:35447)
0620 04:31:55.801438 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:55.801442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:55.805989 (+ 4547us) server_negotiation.cc:407] Connection header received
0620 04:31:55.807308 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:55.807313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:55.807374 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:55.807428 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:55.809604 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:55.810465 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:55.812364 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:55.812718 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:55.812805 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:55.816261 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:55.816266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:55.816268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:31:55.816303 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:55.818114 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:55.818824 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:55.818827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:55.818829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:55.818942 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:55.819404 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:55.819408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:55.819410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:55.819648 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:55.819729 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:55.820528 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 04:31:55.820676 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 04:31:55.822944 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:55.801864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35447 (local address 127.0.98.2:33623)
0620 04:31:55.805742 (+ 3878us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:55.805811 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:55.807002 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:55.808173 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:55.808185 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:55.808572 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:55.809405 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:55.809429 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:55.810650 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:55.810654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:55.812186 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:55.812204 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:55.813105 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:55.813112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:55.813340 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:55.814143 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:55.814165 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:55.815997 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:55.818272 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:55.818280 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:55.818301 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:55.818696 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:55.819058 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:55.819061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:55.819064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:55.819281 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:55.819799 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:55.819806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:55.820221 (+ 415us) client_negotiation.cc:769] Sending connection context
0620 04:31:55.821303 (+ 1082us) client_negotiation.cc:240] Negotiation successful
0620 04:31:55.821685 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3605,"thread_start_us":169,"threads_started":1}
I20260620 04:31:55.824692 11970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35447
I20260620 04:31:55.824983 11970 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:55.825652 11970 heartbeater.cc:511] Master 127.0.98.62:35447 requested a full tablet report, sending...
I20260620 04:31:55.827006 11620 ts_manager.cc:194] Registered new tserver with Master: 47ddafa0e3e1465c9cc65237a61c5aaf (127.0.98.2:40219)
I20260620 04:31:55.827828 11620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33623
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:31:55.852133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42995/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35447
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:31:56.022702 11975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:31:56.023147 11975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:31:56.023263 11975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:31:56.029515 11975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:31:56.029682 11975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:31:56.029824 11975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:31:56.037866 11975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:42995/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:31:56.039552 11975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:31:56.041033 11975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:31:56.073359 11981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:56.074230 11983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:31:56.077282 11980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:31:56.081344 11975 server_base.cc:1034] running on GCE node
I20260620 04:31:56.081974 11975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:31:56.082870 11975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:31:56.098363 11975 hybrid_clock.cc:648] HybridClock initialized: now 1781929916098255 us; error 114 us; skew 500 ppm
Jun 20 04:31:56 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:31:56 dist-test-slave-rzqm krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:31:56.102638 11975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:31:56.103996 11975 webserver.cc:460] Webserver started at http://127.0.98.3:34985/ using document root <none> and password file <none>
I20260620 04:31:56.104905 11975 fs_manager.cc:359] Metadata directory not provided
I20260620 04:31:56.105129 11975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:31:56.105436 11975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:31:56 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:31:56.163818 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "4b1ab6e9e52644c987297e3856b58ee1"
format_stamp: "Formatted at 2026-06-20 04:31:56 on dist-test-slave-rzqm"
server_key: "cd78ff9cce673803b7fe2cce8847eb30"
server_key_iv: "195f2d17278aba0815a34557c78f91ea"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:56.177580 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4b1ab6e9e52644c987297e3856b58ee1"
format_stamp: "Formatted at 2026-06-20 04:31:56 on dist-test-slave-rzqm"
server_key: "cd78ff9cce673803b7fe2cce8847eb30"
server_key_iv: "195f2d17278aba0815a34557c78f91ea"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:56.206812 11975 fs_manager.cc:691] Time spent creating directory manager: real 0.029s user 0.005s sys 0.000s
I20260620 04:31:56.254156 11990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:56.256319 11975 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.004s sys 0.000s
I20260620 04:31:56.256489 11975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "4b1ab6e9e52644c987297e3856b58ee1"
format_stamp: "Formatted at 2026-06-20 04:31:56 on dist-test-slave-rzqm"
server_key: "cd78ff9cce673803b7fe2cce8847eb30"
server_key_iv: "195f2d17278aba0815a34557c78f91ea"
server_key_version: "kuduclusterkey@0"
I20260620 04:31:56.256642 11975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:31:56.288945 11975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:31:56.293207 11975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:31:56.293483 11975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:31:56.294265 11975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:31:56.295471 11975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:31:56.295661 11975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 04:31:56.295758 11975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:31:56.295804 11975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:31:56.306757 11975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45505
I20260620 04:31:56.308163 11975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:31:56.312605 12103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45505 every 8 connection(s)
I20260620 04:31:56.316187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 11975
I20260620 04:31:56.316314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:31:56.316624 392 external_mini_cluster.cc:1428] Setting key e752d5b6e44d12299dd406e4a26dc11a
Jun 20 04:31:56 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:31:56.331071 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.315060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37251 (local address 127.0.98.62:35447)
0620 04:31:56.315229 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:56.315234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:56.317447 (+ 2213us) server_negotiation.cc:407] Connection header received
0620 04:31:56.318746 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:56.318749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:56.318805 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:56.318855 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:56.320731 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.321557 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.323384 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.323735 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.323820 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.326980 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:56.326985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:56.326988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:31:56.327019 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:56.328519 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:56.329181 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:56.329185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:56.329186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:56.329279 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:56.329728 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:56.329732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:56.329733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:56.329974 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:56.330057 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:56.330794 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 04:31:56.330926 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:31:56.333098 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.315628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35447 (local address 127.0.98.3:37251)
0620 04:31:56.317271 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:56.317347 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:56.318510 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:56.319419 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:56.319430 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:56.319807 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:31:56.320551 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.320572 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.321739 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.321742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.323206 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.323219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.324049 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.324052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.324275 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.325044 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:31:56.325067 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:31:56.326748 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:31:56.328659 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:56.328665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:56.328686 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:56.329061 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:56.329408 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:31:56.329411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:31:56.329413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:31:56.329614 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:31:56.330124 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:31:56.330131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:31:56.330507 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 04:31:56.331530 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0620 04:31:56.331921 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"thread_start_us":169,"threads_started":1}
I20260620 04:31:56.334465 12104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35447
I20260620 04:31:56.334769 12104 heartbeater.cc:464] Registering TS with master...
I20260620 04:31:56.335474 12104 heartbeater.cc:511] Master 127.0.98.62:35447 requested a full tablet report, sending...
I20260620 04:31:56.336782 11620 ts_manager.cc:194] Registered new tserver with Master: 4b1ab6e9e52644c987297e3856b58ee1 (127.0.98.3:45505)
I20260620 04:31:56.337558 11620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37251
I20260620 04:31:56.342870 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:31:56.359328 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.345292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.0.98.62:35447)
0620 04:31:56.345423 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:56.345429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:56.349391 (+ 3962us) server_negotiation.cc:407] Connection header received
0620 04:31:56.349592 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:56.349595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:56.349640 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:56.349712 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:31:56.350852 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.351707 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.352991 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.353400 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.353480 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.355514 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:31:56.355552 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:31:56.355555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:31:56.355588 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:31:56.357410 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:56.357950 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:56.357954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:56.357956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:56.358057 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:31:56.358366 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:31:56.358369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:31:56.358371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:31:56.358609 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:31:56.358743 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:56.359011 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 04:31:56.359140 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:31:56.363165 11620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:31:56.367966 11620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:31:56.405620 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.380812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.0.98.2:40219)
0620 04:31:56.381446 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:56.381454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:56.385728 (+ 4274us) server_negotiation.cc:407] Connection header received
0620 04:31:56.386044 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:56.386051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:56.386346 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:56.386483 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:56.387921 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.389195 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.403278 (+ 14083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.404231 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.404341 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.404506 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:56.405199 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 04:31:56.405459 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":85,"threads_started":1}
I20260620 04:31:56.409662 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.381199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.0.98.1:41849)
0620 04:31:56.385257 (+ 4058us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:56.385264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:56.391817 (+ 6553us) server_negotiation.cc:407] Connection header received
0620 04:31:56.391913 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:56.391919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:56.392196 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:56.392367 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:56.392777 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.394005 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.406458 (+ 12453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.407392 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.407511 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.407671 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:56.409373 (+ 1702us) server_negotiation.cc:299] Negotiation successful
0620 04:31:56.409492 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3943,"thread_start_us":73,"threads_started":1}
I20260620 04:31:56.410061 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.380218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41849 (local address 127.0.0.1:55674)
0620 04:31:56.391594 (+ 11376us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:56.391610 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:56.391707 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:56.392526 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:56.392529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:56.392548 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:56.392643 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.392649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.394133 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.394136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.406279 (+ 12143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.406304 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.407749 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.407752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.407857 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.407882 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:31:56.407960 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 04:31:56.408122 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11295,"thread_start_us":108,"threads_started":1}
I20260620 04:31:56.411278 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.381496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.0.98.3:45505)
0620 04:31:56.382290 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:56.382310 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:56.396630 (+ 14320us) server_negotiation.cc:407] Connection header received
0620 04:31:56.396721 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:56.396728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:56.396994 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:56.397190 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:56.398078 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.399270 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.400738 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.402818 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.402935 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.403121 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:56.410997 (+ 7876us) server_negotiation.cc:299] Negotiation successful
0620 04:31:56.411129 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":558,"threads_started":1}
I20260620 04:31:56.410373 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.379920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40219 (local address 127.0.0.1:42670)
0620 04:31:56.385563 (+ 5643us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:56.385631 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:56.385857 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:56.387607 (+ 1750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:56.387613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:56.387640 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:56.387790 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.387796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.389354 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.389357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.397725 (+ 8368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.397736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.404626 (+ 6890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.404630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.404778 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.404835 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 04:31:56.404962 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 04:31:56.409262 (+ 4300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5443,"thread_start_us":121,"threads_started":1}
I20260620 04:31:56.410866 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.380699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45505 (local address 127.0.0.1:53020)
0620 04:31:56.396418 (+ 15719us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:56.396433 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:56.396523 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:56.397859 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:56.397862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:56.397881 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:56.397964 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.397969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.399432 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.399436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.400564 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.400576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.410455 (+ 9879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.410459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.410564 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.410590 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 04:31:56.410667 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 04:31:56.410743 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15642,"thread_start_us":74,"threads_started":1}
I20260620 04:31:56.415458 11885 tablet_service.cc:1467] Processing CreateTablet for tablet 344cb20675f840c89454d13f9634fd6e (DEFAULT_TABLE table=test-table [id=5e01dd817e5544cebf4b28f561200674]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:31:56.417054 11768 tablet_service.cc:1467] Processing CreateTablet for tablet 344cb20675f840c89454d13f9634fd6e (DEFAULT_TABLE table=test-table [id=5e01dd817e5544cebf4b28f561200674]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:31:56.419230 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 344cb20675f840c89454d13f9634fd6e (DEFAULT_TABLE table=test-table [id=5e01dd817e5544cebf4b28f561200674]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:31:56.419881 11768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344cb20675f840c89454d13f9634fd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:31:56.420522 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344cb20675f840c89454d13f9634fd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:31:56.425956 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344cb20675f840c89454d13f9634fd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:31:56.437855 12123 tablet_bootstrap.cc:492] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: Bootstrap starting.
I20260620 04:31:56.440969 12123 tablet_bootstrap.cc:654] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: Neither blocks nor log segments found. Creating new log.
I20260620 04:31:56.441694 12122 tablet_bootstrap.cc:492] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff: Bootstrap starting.
I20260620 04:31:56.442185 12123 log.cc:824] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: Log is configured to *not* fsync() on all Append() calls
I20260620 04:31:56.443221 12124 tablet_bootstrap.cc:492] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: Bootstrap starting.
I20260620 04:31:56.445371 12122 tablet_bootstrap.cc:654] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff: Neither blocks nor log segments found. Creating new log.
I20260620 04:31:56.446192 12124 tablet_bootstrap.cc:654] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: Neither blocks nor log segments found. Creating new log.
I20260620 04:31:56.446348 12122 log.cc:824] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff: Log is configured to *not* fsync() on all Append() calls
I20260620 04:31:56.448518 12123 tablet_bootstrap.cc:492] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: No bootstrap required, opened a new log
I20260620 04:31:56.448711 12122 tablet_bootstrap.cc:492] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff: No bootstrap required, opened a new log
I20260620 04:31:56.448865 12123 ts_tablet_manager.cc:1397] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260620 04:31:56.448936 12122 ts_tablet_manager.cc:1397] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 04:31:56.453230 12123 raft_consensus.cc:348] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "4b1ab6e9e52644c987297e3856b58ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45505 } }
I20260620 04:31:56.453785 12123 raft_consensus.cc:374] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:31:56.453894 12123 raft_consensus.cc:729] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1ab6e9e52644c987297e3856b58ee1, State: Initialized, Role: FOLLOWER
I20260620 04:31:56.454324 12122 raft_consensus.cc:348] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "4b1ab6e9e52644c987297e3856b58ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45505 } }
I20260620 04:31:56.454640 12122 raft_consensus.cc:374] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:31:56.454711 12122 raft_consensus.cc:729] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a70f73d631047bb8995a1105621e7ff, State: Initialized, Role: FOLLOWER
I20260620 04:31:56.455101 12123 consensus_queue.cc:260] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "4b1ab6e9e52644c987297e3856b58ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45505 } }
I20260620 04:31:56.455338 12122 consensus_queue.cc:260] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "4b1ab6e9e52644c987297e3856b58ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45505 } }
I20260620 04:31:56.456135 12124 log.cc:824] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: Log is configured to *not* fsync() on all Append() calls
I20260620 04:31:56.456604 12122 ts_tablet_manager.cc:1428] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 04:31:56.457075 11834 heartbeater.cc:503] Master 127.0.98.62:35447 was elected leader, sending a full tablet report...
I20260620 04:31:56.457297 12123 ts_tablet_manager.cc:1428] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260620 04:31:56.458067 12104 heartbeater.cc:503] Master 127.0.98.62:35447 was elected leader, sending a full tablet report...
I20260620 04:31:56.461241 12124 tablet_bootstrap.cc:492] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: No bootstrap required, opened a new log
I20260620 04:31:56.461555 12124 ts_tablet_manager.cc:1397] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260620 04:31:56.465989 12124 raft_consensus.cc:348] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "4b1ab6e9e52644c987297e3856b58ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45505 } }
I20260620 04:31:56.466379 12124 raft_consensus.cc:374] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:31:56.466498 12124 raft_consensus.cc:729] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ddafa0e3e1465c9cc65237a61c5aaf, State: Initialized, Role: FOLLOWER
I20260620 04:31:56.467077 12124 consensus_queue.cc:260] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "4b1ab6e9e52644c987297e3856b58ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45505 } }
I20260620 04:31:56.467885 11970 heartbeater.cc:503] Master 127.0.98.62:35447 was elected leader, sending a full tablet report...
I20260620 04:31:56.468375 12124 ts_tablet_manager.cc:1428] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 04:31:56.496037 11835 tablet.cc:2369] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:31:56.556119 11971 tablet.cc:2369] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 04:31:56 dist-test-slave-rzqm krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 04:31:56.563552 12105 tablet.cc:2369] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:31:56.589167 12128 raft_consensus.cc:482] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:31:56.589430 12128 raft_consensus.cc:504] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "4b1ab6e9e52644c987297e3856b58ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45505 } }
I20260620 04:31:56.590611 12128 leader_election.cc:288] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47ddafa0e3e1465c9cc65237a61c5aaf (127.0.98.2:40219), 9a70f73d631047bb8995a1105621e7ff (127.0.98.1:41849)
I20260620 04:31:56.595512 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.590824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40219 (local address 127.0.98.3:41041)
0620 04:31:56.590981 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:56.591000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:56.591110 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:56.591719 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:56.591722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:56.591739 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:56.591855 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.591861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.592841 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.592844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.594045 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.594055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.595062 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.595065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.595194 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.595219 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 04:31:56.595287 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:31:56.595364 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 04:31:56.596015 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.591455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58265 (local address 127.0.98.1:41849)
0620 04:31:56.591561 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:56.591565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:56.591579 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:31:56.591636 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:56.591640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:56.591690 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:56.591756 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:56.592438 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.593145 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.594714 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.595304 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.595423 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.595488 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:56.595813 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 04:31:56.595899 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 04:31:56.596279 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.591285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41041 (local address 127.0.98.2:40219)
0620 04:31:56.591413 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:56.591417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:56.591431 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:31:56.591481 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:56.591484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:56.591528 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:56.591593 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:31:56.591975 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.592713 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.594217 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.594823 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.594929 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.595002 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:56.596071 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0620 04:31:56.596160 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:31:56.596477 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.590939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41849 (local address 127.0.98.3:58265)
0620 04:31:56.591247 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:56.591260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:56.591350 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:56.592155 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:31:56.592158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:31:56.592178 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:31:56.592294 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.592300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.593472 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.593475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.594580 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:31:56.594589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.595570 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.595573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:31:56.595666 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.595689 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:31:56.596242 (+ 553us) client_negotiation.cc:240] Negotiation successful
0620 04:31:56.596322 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":119,"threads_started":1}
I20260620 04:31:56.597206 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344cb20675f840c89454d13f9634fd6e" candidate_uuid: "4b1ab6e9e52644c987297e3856b58ee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" is_pre_election: true
I20260620 04:31:56.597198 11788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344cb20675f840c89454d13f9634fd6e" candidate_uuid: "4b1ab6e9e52644c987297e3856b58ee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a70f73d631047bb8995a1105621e7ff" is_pre_election: true
I20260620 04:31:56.597544 11788 raft_consensus.cc:2454] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b1ab6e9e52644c987297e3856b58ee1 in term 0.
I20260620 04:31:56.597544 11924 raft_consensus.cc:2454] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b1ab6e9e52644c987297e3856b58ee1 in term 0.
I20260620 04:31:56.598026 11993 leader_election.cc:302] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b1ab6e9e52644c987297e3856b58ee1, 9a70f73d631047bb8995a1105621e7ff; no voters:
I20260620 04:31:56.598598 12128 raft_consensus.cc:2781] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:31:56.598683 12128 raft_consensus.cc:482] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:31:56.598726 12128 raft_consensus.cc:3037] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:31:56.600052 12128 raft_consensus.cc:504] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "4b1ab6e9e52644c987297e3856b58ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45505 } }
I20260620 04:31:56.600472 12128 leader_election.cc:288] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [CANDIDATE]: Term 1 election: Requested vote from peers 47ddafa0e3e1465c9cc65237a61c5aaf (127.0.98.2:40219), 9a70f73d631047bb8995a1105621e7ff (127.0.98.1:41849)
I20260620 04:31:56.600968 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344cb20675f840c89454d13f9634fd6e" candidate_uuid: "4b1ab6e9e52644c987297e3856b58ee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf"
I20260620 04:31:56.600991 11788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344cb20675f840c89454d13f9634fd6e" candidate_uuid: "4b1ab6e9e52644c987297e3856b58ee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a70f73d631047bb8995a1105621e7ff"
I20260620 04:31:56.601135 11924 raft_consensus.cc:3037] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:31:56.601146 11788 raft_consensus.cc:3037] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:31:56.602137 11788 raft_consensus.cc:2454] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b1ab6e9e52644c987297e3856b58ee1 in term 1.
I20260620 04:31:56.602401 11924 raft_consensus.cc:2454] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b1ab6e9e52644c987297e3856b58ee1 in term 1.
I20260620 04:31:56.602435 11993 leader_election.cc:302] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b1ab6e9e52644c987297e3856b58ee1, 9a70f73d631047bb8995a1105621e7ff; no voters:
I20260620 04:31:56.602619 12128 raft_consensus.cc:2781] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:31:56.602851 12128 raft_consensus.cc:686] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 1 LEADER]: Becoming Leader. State: Replica: 4b1ab6e9e52644c987297e3856b58ee1, State: Running, Role: LEADER
I20260620 04:31:56.603261 12128 consensus_queue.cc:237] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } } peers { permanent_uuid: "4b1ab6e9e52644c987297e3856b58ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45505 } }
I20260620 04:31:56.606900 11620 catalog_manager.cc:5495] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b1ab6e9e52644c987297e3856b58ee1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4b1ab6e9e52644c987297e3856b58ee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b1ab6e9e52644c987297e3856b58ee1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45505 } health_report { overall_health: HEALTHY } } }
I20260620 04:31:56.626186 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.620781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.0.98.3:45505)
0620 04:31:56.620959 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 04:31:56.620964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:31:56.621383 (+ 419us) server_negotiation.cc:407] Connection header received
0620 04:31:56.621481 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:31:56.621485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:31:56.621557 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:31:56.621700 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:31:56.622026 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.622806 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.624037 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:31:56.624371 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:31:56.624459 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:31:56.624901 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:31:56.625507 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:31:56.625639 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 04:31:56.625984 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 04:31:56.626047 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 04:31:56.635963 11788 raft_consensus.cc:1261] T 344cb20675f840c89454d13f9634fd6e P 9a70f73d631047bb8995a1105621e7ff [term 1 FOLLOWER]: Refusing update from remote peer 4b1ab6e9e52644c987297e3856b58ee1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:31:56.636895 12133 consensus_queue.cc:1035] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a70f73d631047bb8995a1105621e7ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:31:56.642432 11924 raft_consensus.cc:1261] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf [term 1 FOLLOWER]: Refusing update from remote peer 4b1ab6e9e52644c987297e3856b58ee1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:31:56.643263 12133 consensus_queue.cc:1035] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47ddafa0e3e1465c9cc65237a61c5aaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:31:56.657232 12137 mvcc.cc:205] Tried to move back new op lower bound from 7298784938532655104 to 7298784938410201088. Current Snapshot: MvccSnapshot[applied={T|T < 7298784938532655104}]
I20260620 04:31:56.677055 11620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:31:56.677265 11620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:31:56.681080 12138 mvcc.cc:205] Tried to move back new op lower bound from 7298784938532655104 to 7298784938410201088. Current Snapshot: MvccSnapshot[applied={T|T < 7298784938532655104}]
I20260620 04:31:56.681185 11620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a512aef8f17d4150b24f85649a522016: Sending DeleteTablet for 3 replicas of tablet 344cb20675f840c89454d13f9634fd6e
I20260620 04:31:56.682152 11885 tablet_service.cc:1514] Processing DeleteTablet for tablet 344cb20675f840c89454d13f9634fd6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:31:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42670
I20260620 04:31:56.683218 11768 tablet_service.cc:1514] Processing DeleteTablet for tablet 344cb20675f840c89454d13f9634fd6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:31:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55674
I20260620 04:31:56.684262 12025 tablet_service.cc:1514] Processing DeleteTablet for tablet 344cb20675f840c89454d13f9634fd6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:31:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53020
I20260620 04:31:56.686002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 11701
I20260620 04:31:56.692056 12140 mvcc.cc:205] Tried to move back new op lower bound from 7298784938532655104 to 7298784938410201088. Current Snapshot: MvccSnapshot[applied={T|T < 7298784938532655104 or (T in {7298784938532655104})}]
I20260620 04:31:56.692979 12146 tablet_replica.cc:330] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: stopping tablet replica
I20260620 04:31:56.693473 12146 raft_consensus.cc:2229] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:31:56.693967 12146 raft_consensus.cc:2258] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 04:31:56.699290 11606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41849 (error 108)
I20260620 04:31:56.700126 12148 tablet_replica.cc:330] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: stopping tablet replica
I20260620 04:31:56.700582 12148 raft_consensus.cc:2229] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:31:56.700677 12146 ts_tablet_manager.cc:1905] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:31:56.701205 12148 raft_consensus.cc:2258] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:31:56.703305 12148 ts_tablet_manager.cc:1905] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:31:56.703799 12146 ts_tablet_manager.cc:1918] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:31:56.703975 12146 log.cc:1196] T 344cb20675f840c89454d13f9634fd6e P 47ddafa0e3e1465c9cc65237a61c5aaf: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/344cb20675f840c89454d13f9634fd6e
I20260620 04:31:56.704160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 11840
I20260620 04:31:56.707299 12148 ts_tablet_manager.cc:1918] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:31:56.707414 12148 log.cc:1196] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/344cb20675f840c89454d13f9634fd6e
I20260620 04:31:56.707742 12148 ts_tablet_manager.cc:1939] T 344cb20675f840c89454d13f9634fd6e P 4b1ab6e9e52644c987297e3856b58ee1: Deleting consensus metadata
I20260620 04:31:56.709064 11606 catalog_manager.cc:4841] TS 4b1ab6e9e52644c987297e3856b58ee1 (127.0.98.3:45505): tablet 344cb20675f840c89454d13f9634fd6e (table test-table [id=5e01dd817e5544cebf4b28f561200674]) successfully deleted
I20260620 04:31:56.709194 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:31:56.708799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41849 (local address 127.0.0.1:55680)
0620 04:31:56.708942 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:56.709076 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 04:31:56.709318 11606 catalog_manager.cc:4584] TS 9a70f73d631047bb8995a1105621e7ff (127.0.98.1:41849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 344cb20675f840c89454d13f9634fd6e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41849: connect: Connection refused (error 111)
W20260620 04:31:56.719655 12116 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:31:56.718931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40219 (local address 127.0.0.1:42680)
0620 04:31:56.719120 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 04:31:56.719135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:31:56.719230 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:31:56.719493 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40219: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 04:31:56.720430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 11975
I20260620 04:31:56.733670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 11588
I20260620 04:31:56.744323 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 04:31:57.090072 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 04:31:57.090209 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 04:31:57.425699 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 04:31:57.425 UTC [11174] LOG: received smart shutdown request
2026-06-20 04:31:57.429 UTC [11174] LOG: background worker "logical replication launcher" (PID 11183) exited with exit code 1
2026-06-20 04:31:57.429 UTC [11178] LOG: shutting down
2026-06-20 04:31:57.601 UTC [11174] LOG: database system is shut down
2026-06-20T04:31:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47200 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:31:57 dist-test-slave-rzqm krb5kdc[12158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:31:57 dist-test-slave-rzqm krb5kdc[12158](info): set up 2 sockets
Jun 20 04:31:57 dist-test-slave-rzqm krb5kdc[12158](info): commencing operation
krb5kdc: starting...
W20260620 04:31:58.648797 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:31:58 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:31:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 04:31:58.815575 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYF3Tc0/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/postgres -l logfile start
2026-06-20 04:32:01.164 UTC [12188] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 04:32:01.165 UTC [12188] LOG: listening on IPv4 address "127.0.98.20", port 53829
2026-06-20 04:32:01.169 UTC [12188] LOG: listening on Unix socket "/tmp/.s.PGSQL.53829"
2026-06-20 04:32:01.197 UTC [12191] LOG: database system was shut down at 2026-06-20 04:31:59 UTC
2026-06-20 04:32:01.201 UTC [12188] LOG: database system is ready to accept connections
I20260620 04:32:02.072685 392 mini_postgres.cc:96] Postgres bound to 53829
2026-06-20 04:32:02.078 UTC [12199] FATAL: database "slave" does not exist
127.0.98.20:53829 - accepting connections
I20260620 04:32:02.079315 392 mini_ranger.cc:161] Starting Ranger...
I20260620 04:32:02.093925 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 04:32:02.511446 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 04:32:02.511511 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-admin
2026-06-20 04:32:02,776 [I] DB FLAVOR :POSTGRES
2026-06-20 04:32:02,776 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:32:02,777 [I] Checking connection..
2026-06-20 04:32:02,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 04:32:03,026 [I] Checking connection passed.
2026-06-20 04:32:03,027 [I] --------- Verifying version history table ---------
2026-06-20 04:32:03,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:32:03,273 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:32:03,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:32:03,514 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 04:32:03,514 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 04:32:03,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 04:32:03.765 UTC [12263] WARNING: there is no transaction in progress
2026-06-20 04:32:03,771 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 04:32:03,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 04:32:04,021 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 04:32:04,021 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:32:04,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 04:32:04,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 04:32:04,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rzqm.c.gcp-upstream.internal','N') ;"
2026-06-20 04:32:04,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:32:04,946 [I] Table x_portal_user does not exist in database ranger
2026-06-20 04:32:04,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:32:05,207 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 04:32:05,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:32:05,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:32:05,678 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 04:32:05,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 04:32:06.582 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.596 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.612 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.627 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.645 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.665 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.734 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.742 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.756 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.767 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.779 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.798 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.807 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.813 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.818 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:06.824 UTC [12371] WARNING: there is no transaction in progress
2026-06-20 04:32:07,411 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 04:32:07,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 04:32:07,647 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 04:32:07,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 04:32:07,892 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 04:32:07,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 04:32:08,157 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 04:32:08,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 04:32:08,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 04:32:08,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rzqm.c.gcp-upstream.internal';"
2026-06-20 04:32:08,863 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 04:32:08,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 04:32:09,095 [I] DB_PATCHES have already been applied
I20260620 04:32:09.100539 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYF3Tc0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 04:32:09.100610 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 04:32:09.103905 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:58007
Jun 20, 2026 4:32:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:32:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:32:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 4:32:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 4:32:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 4:32:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 4:32:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 04:32:10.099048 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.020s user 0.000s sys 0.009s
Jun 20, 2026 4:32:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 4:32:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 4:32:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 4:32:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 4:32:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 04:32:31.381345 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 04:32:31 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929951, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 04:32:31.458762 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 04:32:31.467695 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 04:32:32.219499 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 04:32:32.219581 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-kms
I20260620 04:32:32.219597 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53829
2026-06-20 04:32:32,370 [I] DB FLAVOR :POSTGRES
2026-06-20 04:32:32,370 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 04:32:32,370 [I] Checking connection
2026-06-20 04:32:32,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 04:32:32,604 [I] connection success
2026-06-20 04:32:32,604 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 04:32:32,604 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 04:32:32,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 04:32:32,850 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 04:32:32,850 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 04:32:32,850 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 04:32:32,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53829/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 04:32:33,126 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 04:32:33.569716 392 mini_ranger_kms.cc:310] Created kms service
I20260620 04:32:33.769627 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 04:32:33.812383 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 04:32:33 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929951, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:34.184717 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 04:32:34.184868 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYF3Tc0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYF3Tc0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 04:32:34.184887 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 04:32:34.188676 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12551
I20260620 04:32:34.188746 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60059
Jun 20, 2026 4:32:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 4:32:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 4:32:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 4:32:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 4:32:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 4:32:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 04:32:36.212471 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.754s user 0.019s sys 0.116s
I20260620 04:32:36.212589 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 04:32:36.212646 392 mini_ranger_kms.cc:401] 127.0.98.20:60059/kms/v1/keys
Jun 20 04:32:36 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781929956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 04:32:36 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:32:37 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 04:32:39 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:32:39.576881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39775
--ranger_config_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60059/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:39.684368 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:39.684636 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:39.684687 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:39.688657 12602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:32:39.688727 12602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:39.688762 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:39.688783 12602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:32:39.688802 12602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:32:39.694446 12602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60059/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39775
--ranger_config_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:39.695518 12602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:39.696532 12602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:32:39.705277 12602 server_base.cc:1034] running on GCE node
W20260620 04:32:39.705331 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:39.705341 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:39.705341 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:39.706295 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:39.707509 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:39.708724 12602 hybrid_clock.cc:648] HybridClock initialized: now 1781929959708708 us; error 49 us; skew 500 ppm
Jun 20 04:32:39 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:39.711627 12602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:32:39.712603 12602 webserver.cc:460] Webserver started at http://127.0.98.62:39185/ using document root <none> and password file <none>
I20260620 04:32:39.713181 12602 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:39.713243 12602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:39.713434 12602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:32:39 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:39.774498 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "63e44508bc7a4bf0aa7062acc45a42ed"
format_stamp: "Formatted at 2026-06-20 04:32:39 on dist-test-slave-rzqm"
server_key: "32f7b9be1ed48173fcedf5207a741c7e"
server_key_iv: "4bebf822b3e644369228464f54e6478d"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:39.775087 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "63e44508bc7a4bf0aa7062acc45a42ed"
format_stamp: "Formatted at 2026-06-20 04:32:39 on dist-test-slave-rzqm"
server_key: "32f7b9be1ed48173fcedf5207a741c7e"
server_key_iv: "4bebf822b3e644369228464f54e6478d"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:39.778874 12602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:32:39.793403 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:39.794858 12602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:32:39.795009 12602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "63e44508bc7a4bf0aa7062acc45a42ed"
format_stamp: "Formatted at 2026-06-20 04:32:39 on dist-test-slave-rzqm"
server_key: "32f7b9be1ed48173fcedf5207a741c7e"
server_key_iv: "4bebf822b3e644369228464f54e6478d"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:39.795140 12602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:39.809922 12602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:39.812744 12602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:39.812925 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:39.820413 12602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39775
I20260620 04:32:39.820430 12669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39775 every 8 connection(s)
I20260620 04:32:39.821461 12602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:32:39.824153 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:39.824278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 12602
I20260620 04:32:39.824407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:32:39.824748 392 external_mini_cluster.cc:1428] Setting key 18dd939434feab59d6c7df0a505e3654
I20260620 04:32:39.832671 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: Bootstrap starting.
Jun 20 04:32:39 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:39.835323 12670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:39.836581 12670 log.cc:824] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:39.838781 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: No bootstrap required, opened a new log
I20260620 04:32:39.839120 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:39.826236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.0.98.62:39775)
0620 04:32:39.826637 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:39.826655 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:39.826693 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:32:39.827839 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:39.827858 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:39.828204 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:39.828523 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:39.829377 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:39.830379 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:39.831259 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:39.831746 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:39.831885 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:39.834403 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:39.834408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:39.834422 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:39.834451 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:39.835951 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:39.836366 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:39.836371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:39.836376 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:39.836476 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:39.836756 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:39.836760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:39.836762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:39.837257 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:39.837443 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:39.837778 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 04:32:39.838192 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260620 04:32:39.842196 12670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } }
I20260620 04:32:39.842468 12670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:39.842515 12670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e44508bc7a4bf0aa7062acc45a42ed, State: Initialized, Role: FOLLOWER
I20260620 04:32:39.842924 12670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } }
I20260620 04:32:39.843037 12670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:32:39.843091 12670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:32:39.843182 12670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:39.844059 12670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } }
I20260620 04:32:39.844383 12670 leader_election.cc:302] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e44508bc7a4bf0aa7062acc45a42ed; no voters:
I20260620 04:32:39.844662 12670 leader_election.cc:288] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:32:39.844830 12675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:32:39.845119 12675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 1 LEADER]: Becoming Leader. State: Replica: 63e44508bc7a4bf0aa7062acc45a42ed, State: Running, Role: LEADER
I20260620 04:32:39.845520 12675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } }
I20260620 04:32:39.845881 12670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:32:39.847276 12676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } } }
I20260620 04:32:39.847401 12676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:39.847682 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e44508bc7a4bf0aa7062acc45a42ed. Latest consensus state: current_term: 1 leader_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } } }
I20260620 04:32:39.847744 12685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:32:39.847851 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:39.848157 12670 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 04:32:39.853240 12685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:32:39.865231 12685 catalog_manager.cc:1347] Generated new cluster ID: a9456631d7584758905e9b9bf1096d0c
I20260620 04:32:39.865303 12685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:32:39.874990 12685 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:32:39.876446 12685 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:32:39.885514 12685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: Generated new TSK 0
I20260620 04:32:39.886214 12685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:32:40.555579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39775
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:40.671343 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:40.671612 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:40.671670 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:40.675839 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:40.675910 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:40.675992 12715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:32:40.681568 12715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:40.682644 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:40.683657 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:40.693203 12721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:40.693361 12715 server_base.cc:1034] running on GCE node
W20260620 04:32:40.693233 12720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:40.693238 12723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:40.693959 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:40.694754 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:40.695958 12715 hybrid_clock.cc:648] HybridClock initialized: now 1781929960695929 us; error 38 us; skew 500 ppm
Jun 20 04:32:40 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:40.699587 12715 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:32:40.700819 12715 webserver.cc:460] Webserver started at http://127.0.98.1:40949/ using document root <none> and password file <none>
I20260620 04:32:40.701496 12715 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:40.701586 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:40.701808 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:32:40 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:40.717235 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "087fcf524c4f4de09c1e653e1923806f"
format_stamp: "Formatted at 2026-06-20 04:32:40 on dist-test-slave-rzqm"
server_key: "19705eadd54eecfc59cf31dc93b506f8"
server_key_iv: "5faacc1c0f08d774b841b397d802143a"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:40.718063 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "087fcf524c4f4de09c1e653e1923806f"
format_stamp: "Formatted at 2026-06-20 04:32:40 on dist-test-slave-rzqm"
server_key: "19705eadd54eecfc59cf31dc93b506f8"
server_key_iv: "5faacc1c0f08d774b841b397d802143a"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:40.722321 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 04:32:40.738905 12732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:40.740727 12715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:32:40.740876 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "087fcf524c4f4de09c1e653e1923806f"
format_stamp: "Formatted at 2026-06-20 04:32:40 on dist-test-slave-rzqm"
server_key: "19705eadd54eecfc59cf31dc93b506f8"
server_key_iv: "5faacc1c0f08d774b841b397d802143a"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:40.741047 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:40.760468 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:40.764034 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:40.764235 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:40.764963 12715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:40.766072 12715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:40.766157 12715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:40.766224 12715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:40.766259 12715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:40.776309 12715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44963
I20260620 04:32:40.777433 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:32:40.784003 12845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44963 every 8 connection(s)
I20260620 04:32:40.786471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 12715
I20260620 04:32:40.786587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:32:40.786891 392 external_mini_cluster.cc:1428] Setting key 335a7487ff64c6d673e51bf6b99f2cd2
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 20 04:32:40 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:40.804136 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:40.785573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35587 (local address 127.0.98.62:39775)
0620 04:32:40.785885 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:40.785897 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:40.790386 (+ 4489us) server_negotiation.cc:407] Connection header received
0620 04:32:40.791629 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:40.791633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:40.791698 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:40.791777 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:40.793352 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:40.794210 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:40.795959 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:40.796318 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:40.796397 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:40.799427 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:40.799431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:40.799434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:40.799463 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:40.801558 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:40.802213 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:40.802217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:40.802219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:40.802317 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:40.802747 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:40.802750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:40.802752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:40.802990 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:40.803075 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:40.803800 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 04:32:40.803938 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":119,"threads_started":1}
I20260620 04:32:40.805754 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:40.786111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39775 (local address 127.0.98.1:35587)
0620 04:32:40.788715 (+ 2604us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:40.790272 (+ 1557us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:40.791414 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:40.792087 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:40.792096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:40.792439 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:40.793175 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:40.793199 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:40.794378 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:40.794381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:40.795816 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:40.795828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:40.796617 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:40.796621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:40.796837 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:40.797533 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:40.797553 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:40.799227 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:40.801711 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:40.801718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:40.801737 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:40.802092 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:40.802429 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:40.802432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:40.802434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:40.802620 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:40.803143 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:40.803150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:40.803499 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 04:32:40.804299 (+ 800us) client_negotiation.cc:240] Negotiation successful
0620 04:32:40.804647 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2356,"thread_start_us":139,"threads_started":1}
I20260620 04:32:40.807127 12846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39775
I20260620 04:32:40.807374 12846 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:40.807864 12846 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
I20260620 04:32:40.809391 12634 ts_manager.cc:194] Registered new tserver with Master: 087fcf524c4f4de09c1e653e1923806f (127.0.98.1:44963)
I20260620 04:32:40.810562 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:32:40.874102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39775
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:32:40 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781929960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 04:32:41.032752 12854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:41.033144 12854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:41.033236 12854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:41.039426 12854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:41.039513 12854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:41.039600 12854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:32:41.047735 12854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:41.049203 12854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:41.050577 12854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:41.063072 12861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:41.063551 12860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:41.063898 12854 server_base.cc:1034] running on GCE node
W20260620 04:32:41.064116 12863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:41.064550 12854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:41.065294 12854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:41.068801 12854 hybrid_clock.cc:648] HybridClock initialized: now 1781929961068725 us; error 94 us; skew 500 ppm
Jun 20 04:32:41 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:41.071887 12854 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:32:41.073124 12854 webserver.cc:460] Webserver started at http://127.0.98.2:36803/ using document root <none> and password file <none>
I20260620 04:32:41.073939 12854 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:41.074033 12854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:41.074271 12854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:32:41 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:41.098410 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "80c690e61c0b4df48f823263c68ce68f"
format_stamp: "Formatted at 2026-06-20 04:32:41 on dist-test-slave-rzqm"
server_key: "b1540d17d2bd9da7b21711b60d878937"
server_key_iv: "0b99a1027fb809ac5168bf2c2b37ff30"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:41.099386 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80c690e61c0b4df48f823263c68ce68f"
format_stamp: "Formatted at 2026-06-20 04:32:41 on dist-test-slave-rzqm"
server_key: "b1540d17d2bd9da7b21711b60d878937"
server_key_iv: "0b99a1027fb809ac5168bf2c2b37ff30"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:41.113700 12854 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.000s
I20260620 04:32:41.133180 12870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:41.134936 12854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:32:41.135082 12854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "80c690e61c0b4df48f823263c68ce68f"
format_stamp: "Formatted at 2026-06-20 04:32:41 on dist-test-slave-rzqm"
server_key: "b1540d17d2bd9da7b21711b60d878937"
server_key_iv: "0b99a1027fb809ac5168bf2c2b37ff30"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:41.135226 12854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:41.149916 12854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:41.158521 12854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:41.158756 12854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:41.159507 12854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:41.160596 12854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:41.160669 12854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:41.160780 12854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:41.160813 12854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:41.174541 12854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33151
I20260620 04:32:41.175791 12854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:32:41.177423 12983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33151 every 8 connection(s)
I20260620 04:32:41.181844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 12854
I20260620 04:32:41.181967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:32:41.182286 392 external_mini_cluster.cc:1428] Setting key 9b7e273df897b78d983d3b9c27ada31d
Jun 20 04:32:41 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:41.203254 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:41.181870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57957 (local address 127.0.98.62:39775)
0620 04:32:41.182009 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:41.182014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:41.182561 (+ 547us) server_negotiation.cc:407] Connection header received
0620 04:32:41.183888 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:41.183891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:41.183949 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:41.183999 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:41.185705 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:41.186521 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:41.188236 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:41.188590 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:41.188660 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:41.192264 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:41.192268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:41.192271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:41.192302 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:41.194059 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:41.199229 (+ 5170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:41.199234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:41.199236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:41.199360 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:41.201296 (+ 1936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:41.201301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:41.201303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:41.201564 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:41.201663 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:41.202886 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0620 04:32:41.203057 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:32:41.204773 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:41.180613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39775 (local address 127.0.98.2:57957)
0620 04:32:41.182381 (+ 1768us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:41.182444 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:41.183655 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:41.184475 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:41.184485 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:41.184830 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:41.185532 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:41.185550 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:41.186682 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:41.186685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:41.188097 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:41.188108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:41.188859 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:41.188862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:41.189093 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:41.190242 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:41.190264 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:41.192065 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:41.198428 (+ 6363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:41.198438 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:41.198461 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:41.199037 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:41.200819 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:41.200823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:41.200855 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:41.201161 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:41.202120 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:41.202130 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:41.202559 (+ 429us) client_negotiation.cc:769] Sending connection context
0620 04:32:41.202804 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 04:32:41.203187 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1542,"thread_start_us":163,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:32:41.208117 12984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39775
I20260620 04:32:41.208405 12984 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:41.209100 12984 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
I20260620 04:32:41.210331 12634 ts_manager.cc:194] Registered new tserver with Master: 80c690e61c0b4df48f823263c68ce68f (127.0.98.2:33151)
I20260620 04:32:41.211194 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:32:41.241796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39775
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:41.420226 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:41.420578 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:41.420636 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:41.426756 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:41.426833 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:41.426918 12989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:32:41.435266 12989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:41.436615 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:41.438206 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 04:32:41 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 04:32:41.451196 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:41.451995 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:41.452690 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:41.452972 12989 server_base.cc:1034] running on GCE node
I20260620 04:32:41.453500 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:41.454276 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:41.455452 12989 hybrid_clock.cc:648] HybridClock initialized: now 1781929961455384 us; error 82 us; skew 500 ppm
Jun 20 04:32:41 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:41.458925 12989 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:32:41.460003 12989 webserver.cc:460] Webserver started at http://127.0.98.3:38373/ using document root <none> and password file <none>
I20260620 04:32:41.460683 12989 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:41.460763 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:41.460983 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 04:32:41 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:41.480014 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "d4665721111e4ef890426feeb623d62a"
format_stamp: "Formatted at 2026-06-20 04:32:41 on dist-test-slave-rzqm"
server_key: "44ed91e54e014daf6e09a106239fc843"
server_key_iv: "f9b05532267c9c1423abc106a856fed3"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:41.480716 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d4665721111e4ef890426feeb623d62a"
format_stamp: "Formatted at 2026-06-20 04:32:41 on dist-test-slave-rzqm"
server_key: "44ed91e54e014daf6e09a106239fc843"
server_key_iv: "f9b05532267c9c1423abc106a856fed3"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:41.486600 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 04:32:41.511667 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:41.514441 12989 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260620 04:32:41.514614 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "d4665721111e4ef890426feeb623d62a"
format_stamp: "Formatted at 2026-06-20 04:32:41 on dist-test-slave-rzqm"
server_key: "44ed91e54e014daf6e09a106239fc843"
server_key_iv: "f9b05532267c9c1423abc106a856fed3"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:41.514777 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:41.558295 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:41.562095 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:41.562388 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:41.563151 12989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:41.564250 12989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:41.564358 12989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:41.564452 12989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:41.564510 12989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:41.574903 12989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32951
I20260620 04:32:41.576148 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:32:41.581351 13117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32951 every 8 connection(s)
I20260620 04:32:41.582083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 12989
I20260620 04:32:41.582195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:32:41.582504 392 external_mini_cluster.cc:1428] Setting key 6ec7bbcf642b678544238b2c09b5e269
Jun 20 04:32:41 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:41.599534 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:41.584737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53471 (local address 127.0.98.62:39775)
0620 04:32:41.584855 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:41.584859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:41.586656 (+ 1797us) server_negotiation.cc:407] Connection header received
0620 04:32:41.587876 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:41.587880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:41.587935 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:41.587988 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:41.589631 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:41.590455 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:41.592177 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:41.592401 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:41.592492 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:41.595399 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:41.595404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:41.595407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:41.595438 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:41.597120 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:41.597749 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:41.597753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:41.597755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:41.597839 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:41.598243 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:41.598246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:41.598248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:41.598466 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:41.598539 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:41.599245 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 04:32:41.599368 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:32:41.601166 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:41.585304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39775 (local address 127.0.98.3:53471)
0620 04:32:41.586509 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:41.586566 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:41.587628 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:41.588389 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:41.588398 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:41.588773 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:41.589461 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:41.589479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:41.590640 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:41.590644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:41.592042 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:41.592051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:41.592611 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:41.592615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:41.592782 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:41.593453 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:41.593474 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:41.595178 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:41.597273 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:41.597279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:41.597299 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:41.597637 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:41.597937 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:41.597940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:41.597942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:41.598128 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:41.598603 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:41.598610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:41.598958 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 04:32:41.599714 (+ 756us) client_negotiation.cc:240] Negotiation successful
0620 04:32:41.600078 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":148,"threads_started":1}
I20260620 04:32:41.602506 13118 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39775
I20260620 04:32:41.602761 13118 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:41.603336 13118 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
I20260620 04:32:41.604461 12634 ts_manager.cc:194] Registered new tserver with Master: d4665721111e4ef890426feeb623d62a (127.0.98.3:32951)
I20260620 04:32:41.605258 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53471
I20260620 04:32:41.609160 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:32:41.609773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 12715
I20260620 04:32:41.615026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 12854
I20260620 04:32:41.620215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 12989
I20260620 04:32:41.625929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 12602
I20260620 04:32:41.631464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39775
--webserver_interface=127.0.98.62
--webserver_port=39185
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39775
--ranger_config_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60059/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60059/kms with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:41.739657 13121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:41.739926 13121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:41.739965 13121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:41.744040 13121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:32:41.744112 13121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:41.744130 13121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:41.744146 13121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:32:41.744160 13121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:32:41.749910 13121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60059/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39775
--ranger_config_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:41.751058 13121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:41.752183 13121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:41.760660 13129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:41.760673 13126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:41.760812 13121 server_base.cc:1034] running on GCE node
W20260620 04:32:41.760677 13127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:41.761512 13121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:41.762627 13121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:41.763795 13121 hybrid_clock.cc:648] HybridClock initialized: now 1781929961763771 us; error 40 us; skew 500 ppm
Jun 20 04:32:41 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:41.766535 13121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:32:41.767498 13121 webserver.cc:460] Webserver started at http://127.0.98.62:39185/ using document root <none> and password file <none>
I20260620 04:32:41.768031 13121 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:41.768105 13121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:32:41 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:41.814390 13121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:32:41.816653 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:41.818171 13121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:32:41.818293 13121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "63e44508bc7a4bf0aa7062acc45a42ed"
format_stamp: "Formatted at 2026-06-20 04:32:39 on dist-test-slave-rzqm"
server_key: "32f7b9be1ed48173fcedf5207a741c7e"
server_key_iv: "4bebf822b3e644369228464f54e6478d"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:41.818835 13121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:41.842522 13121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:41.845460 13121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:41.845639 13121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:41.853065 13121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39775
I20260620 04:32:41.853127 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39775 every 8 connection(s)
I20260620 04:32:41.854169 13121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:32:41.858357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 13121
I20260620 04:32:41.858469 13190 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:32:41.858835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44963
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39775
--webserver_port=40949
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60059/kms with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:32:41.860541 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: Bootstrap starting.
I20260620 04:32:41.882308 13190 log.cc:824] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:41.890973 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:32:41.891419 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: Bootstrap complete.
I20260620 04:32:41.896796 13190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } }
I20260620 04:32:41.897328 13190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e44508bc7a4bf0aa7062acc45a42ed, State: Initialized, Role: FOLLOWER
I20260620 04:32:41.897894 13190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } }
I20260620 04:32:41.898036 13190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:32:41.898097 13190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:32:41.898236 13190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:32:41.900761 13190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } }
I20260620 04:32:41.901196 13190 leader_election.cc:302] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e44508bc7a4bf0aa7062acc45a42ed; no voters:
I20260620 04:32:41.901563 13190 leader_election.cc:288] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:32:41.901785 13194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:32:41.902271 13194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [term 2 LEADER]: Becoming Leader. State: Replica: 63e44508bc7a4bf0aa7062acc45a42ed, State: Running, Role: LEADER
I20260620 04:32:41.902781 13194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } }
I20260620 04:32:41.903084 13190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:32:41.903995 13194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } } }
I20260620 04:32:41.904201 13194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:41.904443 13194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e44508bc7a4bf0aa7062acc45a42ed. Latest consensus state: current_term: 2 leader_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e44508bc7a4bf0aa7062acc45a42ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39775 } } }
I20260620 04:32:41.904574 13194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:41.904865 13203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:32:41.905305 13190 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 04:32:41.911201 13203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:32:41.912187 13203 catalog_manager.cc:1259] Loaded cluster ID: a9456631d7584758905e9b9bf1096d0c
I20260620 04:32:41.913038 13203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:32:41.915920 13203 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:32:41.916667 13203 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: Loaded TSK: 0
I20260620 04:32:41.918107 13203 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:32:42.026381 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:42.026786 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:42.026865 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:42.033432 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:42.033519 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:42.033619 13192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:32:42.041399 13192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40949
--tserver_master_addrs=127.0.98.62:39775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:42.042766 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:42.044178 13192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:42.058910 13228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:42.058998 13227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:42.059404 13192 server_base.cc:1034] running on GCE node
W20260620 04:32:42.059971 13230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:42.060477 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:42.061365 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:42.062572 13192 hybrid_clock.cc:648] HybridClock initialized: now 1781929962062535 us; error 52 us; skew 500 ppm
Jun 20 04:32:42 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:42.069208 13192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:32:42.070385 13192 webserver.cc:460] Webserver started at http://127.0.98.1:40949/ using document root <none> and password file <none>
I20260620 04:32:42.071182 13192 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:42.071287 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:32:42 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:42.096158 13192 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260620 04:32:42.101169 13238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:42.102834 13192 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 04:32:42.102972 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "087fcf524c4f4de09c1e653e1923806f"
format_stamp: "Formatted at 2026-06-20 04:32:40 on dist-test-slave-rzqm"
server_key: "19705eadd54eecfc59cf31dc93b506f8"
server_key_iv: "5faacc1c0f08d774b841b397d802143a"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:42.103763 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:42.138711 13192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:42.142618 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:42.142836 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:42.143589 13192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:42.144697 13192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:42.144765 13192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:42.144824 13192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:42.144853 13192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:42.156081 13192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44963
I20260620 04:32:42.157773 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:32:42.158463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 13192
I20260620 04:32:42.158891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33151
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39775
--webserver_port=36803
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60059/kms with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:32:42.178184 13351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44963 every 8 connection(s)
Jun 20 04:32:42 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:42.201480 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:42.164643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49205 (local address 127.0.98.62:39775)
0620 04:32:42.165296 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:42.165321 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:42.173091 (+ 7770us) server_negotiation.cc:407] Connection header received
0620 04:32:42.174459 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:42.174474 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:42.174947 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:42.175306 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:42.179992 (+ 4686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.181559 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.189201 (+ 7642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.190120 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.190308 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:42.195365 (+ 5057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:42.195373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:42.195394 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:42.195437 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:42.197611 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:42.198322 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:42.198329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:42.198335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:42.198489 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:42.198946 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:42.198950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:42.198951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:42.199474 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:42.199599 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:42.200251 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 04:32:42.200504 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":116,"threads_started":1}
I20260620 04:32:42.201490 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:42.163014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39775 (local address 127.0.98.1:49205)
0620 04:32:42.171852 (+ 8838us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:42.171930 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:42.173797 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:42.178691 (+ 4894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:42.178702 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:42.179049 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:42.179764 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:42.179785 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.183727 (+ 3942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.183731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:42.185313 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:42.185328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.190688 (+ 5360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.190692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:42.190911 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:42.193249 (+ 2338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:42.193273 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:42.195086 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:42.197780 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:42.197787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:42.197804 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:42.198161 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:42.198629 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:42.198632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:42.198634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:42.198833 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:42.199656 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:42.199661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:42.199911 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 04:32:42.200079 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 04:32:42.200430 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8533,"thread_start_us":149,"threads_started":1}
I20260620 04:32:42.204850 13352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39775
I20260620 04:32:42.205128 13352 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:42.205744 13352 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
W20260620 04:32:42.208102 13352 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39775 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 04:32:42.208226 13352 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:42.208416 13352 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
I20260620 04:32:42.209338 13352 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:42.209530 13352 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
W20260620 04:32:42.210307 13352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:32:42.329375 13354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:42.329654 13354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:42.329718 13354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:42.334467 13354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:42.334547 13354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:42.334636 13354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:32:42.340741 13354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36803
--tserver_master_addrs=127.0.98.62:39775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:42.342113 13354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:42.343380 13354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:42.356070 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:42.356141 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:42.356071 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:42.356551 13354 server_base.cc:1034] running on GCE node
I20260620 04:32:42.356990 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:42.357803 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:42.358982 13354 hybrid_clock.cc:648] HybridClock initialized: now 1781929962358965 us; error 37 us; skew 500 ppm
Jun 20 04:32:42 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:42.361966 13354 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:32:42.362986 13354 webserver.cc:460] Webserver started at http://127.0.98.2:36803/ using document root <none> and password file <none>
I20260620 04:32:42.363736 13354 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:42.363848 13354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:32:42 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:42.385831 13354 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 04:32:42.388912 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:42.390502 13354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:32:42.390650 13354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "80c690e61c0b4df48f823263c68ce68f"
format_stamp: "Formatted at 2026-06-20 04:32:41 on dist-test-slave-rzqm"
server_key: "b1540d17d2bd9da7b21711b60d878937"
server_key_iv: "0b99a1027fb809ac5168bf2c2b37ff30"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:42.391377 13354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:42.407935 13354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:42.411708 13354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:42.411913 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:42.412603 13354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:42.413642 13354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:42.413707 13354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:42.413764 13354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:42.413794 13354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:42.425796 13354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33151
I20260620 04:32:42.427155 13354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:32:42.428179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 13354
I20260620 04:32:42.428649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32951
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39775
--webserver_port=38373
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60059/kms with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:32:42.431155 13485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33151 every 8 connection(s)
Jun 20 04:32:42 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:42.449180 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:42.434708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50307 (local address 127.0.98.62:39775)
0620 04:32:42.434832 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:42.434837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:42.437077 (+ 2240us) server_negotiation.cc:407] Connection header received
0620 04:32:42.438075 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:42.438079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:42.438137 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:42.438195 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:42.439561 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.440282 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.441597 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.441820 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.441907 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:42.444739 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:42.444743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:42.444746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:42.444779 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:42.446481 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:42.447166 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:42.447170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:42.447172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:42.447264 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:42.447694 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:42.447697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:42.447699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:42.447949 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:42.448028 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:42.448823 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 04:32:42.448979 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:32:42.451308 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:42.433567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39775 (local address 127.0.98.2:50307)
0620 04:32:42.436871 (+ 3304us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:42.436938 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:42.437844 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:42.438616 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:42.438639 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:42.438926 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:42.439401 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:42.439414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.440454 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.440457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:42.441464 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:42.441474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.441981 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.441985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:42.442177 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:42.442806 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:42.442826 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:42.444527 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:42.446629 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:42.446636 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:42.446653 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:42.447042 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:42.447380 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:42.447383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:42.447385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:42.447579 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:42.448114 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:42.448122 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:42.448493 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 04:32:42.449708 (+ 1215us) client_negotiation.cc:240] Negotiation successful
0620 04:32:42.450117 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3075,"thread_start_us":134,"threads_started":1}
I20260620 04:32:42.452890 13486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39775
I20260620 04:32:42.453264 13486 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:42.453867 13486 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
W20260620 04:32:42.455859 13486 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39775 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 04:32:42.455991 13486 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:42.456158 13486 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
I20260620 04:32:42.456988 13486 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:42.457173 13486 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
W20260620 04:32:42.457880 13486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:32:42.583722 13488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:42.583982 13488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:42.584044 13488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:42.588301 13488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:42.588366 13488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:42.588450 13488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:32:42.593833 13488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38373
--tserver_master_addrs=127.0.98.62:39775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:42.594908 13488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:42.595960 13488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:42.606274 13496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:42.606487 13498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:42.606710 13495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:42.606875 13488 server_base.cc:1034] running on GCE node
I20260620 04:32:42.607327 13488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:42.608183 13488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:42.609369 13488 hybrid_clock.cc:648] HybridClock initialized: now 1781929962609340 us; error 45 us; skew 500 ppm
Jun 20 04:32:42 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:42.619021 13488 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:32:42.620122 13488 webserver.cc:460] Webserver started at http://127.0.98.3:38373/ using document root <none> and password file <none>
I20260620 04:32:42.620702 13488 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:42.620785 13488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 04:32:42 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:42.638887 13488 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260620 04:32:42.642762 13506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:42.644351 13488 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:32:42.644549 13488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "d4665721111e4ef890426feeb623d62a"
format_stamp: "Formatted at 2026-06-20 04:32:41 on dist-test-slave-rzqm"
server_key: "44ed91e54e014daf6e09a106239fc843"
server_key_iv: "f9b05532267c9c1423abc106a856fed3"
server_key_version: "kuduclusterkey@0"
I20260620 04:32:42.645256 13488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:42.666869 13488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:42.670527 13488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:42.670754 13488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:42.671532 13488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:42.672394 13488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:42.672441 13488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:42.672479 13488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:42.672493 13488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:42.682343 13488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32951
I20260620 04:32:42.682384 13619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32951 every 8 connection(s)
I20260620 04:32:42.683358 13488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:32:42.688673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 13488
Jun 20 04:32:42 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:42.700376 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:42.685891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38745 (local address 127.0.98.62:39775)
0620 04:32:42.686024 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:42.686029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:42.687665 (+ 1636us) server_negotiation.cc:407] Connection header received
0620 04:32:42.688971 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:42.688975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:42.689059 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:42.689157 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:42.691020 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.691752 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.692938 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.693321 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.693427 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:42.696168 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:42.696172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:42.696174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:42.696203 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:42.697909 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:42.698594 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:42.698597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:42.698599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:42.698696 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:42.699140 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:42.699143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:42.699145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:42.699360 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:42.699451 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:42.700151 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 04:32:42.700252 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:32:42.701597 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:42.685979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39775 (local address 127.0.98.3:38745)
0620 04:32:42.687478 (+ 1499us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:42.687548 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:42.688694 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:42.689702 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:42.689713 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:42.690071 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:42.690831 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:42.690854 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.691880 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.691883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:42.692804 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:42.692814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.693604 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.693608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:42.693823 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:42.694469 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:42.694490 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:42.695984 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:42.698061 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:42.698068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:42.698088 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:42.698455 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:42.698815 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:42.698818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:42.698820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:42.699020 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:42.699515 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:42.699523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:42.699884 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 04:32:42.700109 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 04:32:42.700473 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"thread_start_us":152,"threads_started":1}
I20260620 04:32:42.702859 13620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39775
I20260620 04:32:42.703128 13620 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:42.703914 13620 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
W20260620 04:32:42.705669 13620 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39775 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 04:32:42.705802 13620 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:42.706001 13620 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
I20260620 04:32:42.706804 13620 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:42.706979 13620 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
W20260620 04:32:42.707700 13620 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:32:42.708245 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:42.689351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.0.98.62:39775)
0620 04:32:42.689603 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:42.689626 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:42.691312 (+ 1686us) server_negotiation.cc:407] Connection header received
0620 04:32:42.691407 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:42.691410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:42.691459 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:42.691546 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:42.696044 (+ 4498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.697411 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.699049 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:42.699360 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:42.699461 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:42.703988 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:42.703993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:42.703996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:42.704022 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:42.705908 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:42.706416 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:42.706420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:42.706422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:42.706528 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:42.706850 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:42.706854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:42.706855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:42.707075 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:42.707269 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:42.707949 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 04:32:42.708077 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
Jun 20 04:32:43 dist-test-slave-rzqm krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781929963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:43.210572 13352 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:43.210863 13352 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
I20260620 04:32:43.211956 13154 ts_manager.cc:194] Registered new tserver with Master: 087fcf524c4f4de09c1e653e1923806f (127.0.98.1:44963)
I20260620 04:32:43.213682 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49205
I20260620 04:32:43.458145 13486 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:43.458495 13486 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
I20260620 04:32:43.459574 13154 ts_manager.cc:194] Registered new tserver with Master: 80c690e61c0b4df48f823263c68ce68f (127.0.98.2:33151)
I20260620 04:32:43.460417 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50307
Jun 20 04:32:43 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:43.707962 13620 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:43.708272 13620 heartbeater.cc:511] Master 127.0.98.62:39775 requested a full tablet report, sending...
I20260620 04:32:43.709268 13154 ts_manager.cc:194] Registered new tserver with Master: d4665721111e4ef890426feeb623d62a (127.0.98.3:32951)
I20260620 04:32:43.710040 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38745
I20260620 04:32:43.713223 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:32:43.725219 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:43.714919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.0.98.62:39775)
0620 04:32:43.715207 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:43.715211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:43.715341 (+ 130us) server_negotiation.cc:407] Connection header received
0620 04:32:43.715450 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:43.715453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:43.715504 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:43.715593 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:43.716469 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.717303 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.718474 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.718836 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.718943 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:43.720675 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:43.720680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:43.720684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:43.720721 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:43.722759 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:43.723411 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:43.723415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:43.723417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:43.723513 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:43.723819 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:43.723823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:43.723824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:43.724060 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:43.724206 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:43.724845 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 04:32:43.724991 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
I20260620 04:32:43.729485 13154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:32:43.734488 13154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:32:43.759286 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:43.750567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.0.98.1:44963)
0620 04:32:43.751854 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:43.751862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:43.751885 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:32:43.751958 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:43.751963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:43.752177 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:43.752313 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:43.754388 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.755609 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.757311 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.758207 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.758319 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:43.758502 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:43.758993 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 04:32:43.759124 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181,"thread_start_us":50,"threads_started":1}
I20260620 04:32:43.760234 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:43.750463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44963 (local address 127.0.0.1:43152)
0620 04:32:43.751385 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:43.751451 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:43.751578 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:43.752851 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:43.752857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:43.752881 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:43.753047 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:43.753055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.755746 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.755749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:43.757163 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:43.757175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.758576 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.758580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:43.758743 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:43.758792 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 04:32:43.758916 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 04:32:43.760087 (+ 1171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":32,"thread_start_us":136,"threads_started":1}
I20260620 04:32:43.762938 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:43.751452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33151 (local address 127.0.0.1:53132)
0620 04:32:43.754101 (+ 2649us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:43.754116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:43.754205 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:43.755118 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:43.755121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:43.755139 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:43.755239 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:43.755246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.757630 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.757634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:43.759883 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:43.759893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.762507 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.762511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:43.762606 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:43.762632 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 04:32:43.762706 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 04:32:43.762783 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1784,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260620 04:32:43.763701 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:43.751858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.0.98.2:33151)
0620 04:32:43.754439 (+ 2581us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:43.754445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:43.754471 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:32:43.754549 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:43.754556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:43.754784 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:43.754939 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:43.755374 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.757497 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.760204 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.762186 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.762293 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:43.762440 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:43.763438 (+ 998us) server_negotiation.cc:299] Negotiation successful
0620 04:32:43.763556 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2474,"thread_start_us":87,"threads_started":1}
I20260620 04:32:43.763772 13267 tablet_service.cc:1467] Processing CreateTablet for tablet f1720df409c5413fa347b41f08df1527 (DEFAULT_TABLE table=test-table [id=80dc62e6fbe646a29da6b41c7565fff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:32:43.764988 13267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1720df409c5413fa347b41f08df1527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:43.766994 13420 tablet_service.cc:1467] Processing CreateTablet for tablet f1720df409c5413fa347b41f08df1527 (DEFAULT_TABLE table=test-table [id=80dc62e6fbe646a29da6b41c7565fff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:32:43.768086 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1720df409c5413fa347b41f08df1527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:43.779511 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:43.752613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32951 (local address 127.0.0.1:50868)
0620 04:32:43.759164 (+ 6551us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:43.759179 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:43.759353 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:43.772392 (+ 13039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:43.772395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:43.772430 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:43.772545 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:43.772553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.774399 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.774402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:43.775549 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:43.775562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.779048 (+ 3486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.779052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:43.779165 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:43.779193 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 04:32:43.779266 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 04:32:43.779355 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6480,"thread_start_us":91,"threads_started":1}
I20260620 04:32:43.780102 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:43.752764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.0.98.3:32951)
0620 04:32:43.753050 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:43.753056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:43.759276 (+ 6220us) server_negotiation.cc:407] Connection header received
0620 04:32:43.759511 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:43.759517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:43.759752 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:43.759874 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:43.772701 (+ 12827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.774130 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.777765 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:43.778718 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:43.778821 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:43.778979 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:43.779843 (+ 864us) server_negotiation.cc:299] Negotiation successful
0620 04:32:43.779959 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260620 04:32:43.780236 13654 tablet_bootstrap.cc:492] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: Bootstrap starting.
I20260620 04:32:43.780671 13655 tablet_bootstrap.cc:492] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f: Bootstrap starting.
I20260620 04:32:43.783746 13554 tablet_service.cc:1467] Processing CreateTablet for tablet f1720df409c5413fa347b41f08df1527 (DEFAULT_TABLE table=test-table [id=80dc62e6fbe646a29da6b41c7565fff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:32:43.784917 13554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1720df409c5413fa347b41f08df1527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:43.794047 13655 tablet_bootstrap.cc:654] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:43.795107 13655 log.cc:824] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:43.796010 13654 tablet_bootstrap.cc:654] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:43.796502 13656 tablet_bootstrap.cc:492] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: Bootstrap starting.
I20260620 04:32:43.798411 13654 log.cc:824] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:43.798625 13655 tablet_bootstrap.cc:492] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f: No bootstrap required, opened a new log
I20260620 04:32:43.798838 13655 ts_tablet_manager.cc:1397] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260620 04:32:43.799319 13656 tablet_bootstrap.cc:654] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:43.800230 13656 log.cc:824] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:43.802476 13656 tablet_bootstrap.cc:492] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: No bootstrap required, opened a new log
I20260620 04:32:43.802712 13656 ts_tablet_manager.cc:1397] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:32:43.803254 13655 raft_consensus.cc:348] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 } } peers { permanent_uuid: "80c690e61c0b4df48f823263c68ce68f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33151 } } peers { permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } }
I20260620 04:32:43.803560 13655 raft_consensus.cc:374] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:43.803627 13655 raft_consensus.cc:729] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087fcf524c4f4de09c1e653e1923806f, State: Initialized, Role: FOLLOWER
I20260620 04:32:43.804418 13655 consensus_queue.cc:260] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 } } peers { permanent_uuid: "80c690e61c0b4df48f823263c68ce68f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33151 } } peers { permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } }
I20260620 04:32:43.805626 13655 ts_tablet_manager.cc:1428] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260620 04:32:43.806068 13352 heartbeater.cc:503] Master 127.0.98.62:39775 was elected leader, sending a full tablet report...
I20260620 04:32:43.807629 13656 raft_consensus.cc:348] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 } } peers { permanent_uuid: "80c690e61c0b4df48f823263c68ce68f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33151 } } peers { permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } }
I20260620 04:32:43.807929 13656 raft_consensus.cc:374] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:43.807984 13656 raft_consensus.cc:729] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4665721111e4ef890426feeb623d62a, State: Initialized, Role: FOLLOWER
I20260620 04:32:43.808166 13654 tablet_bootstrap.cc:492] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: No bootstrap required, opened a new log
I20260620 04:32:43.808401 13654 ts_tablet_manager.cc:1397] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
Jun 20 04:32:43 dist-test-slave-rzqm krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 04:32:43.809367 13656 consensus_queue.cc:260] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 } } peers { permanent_uuid: "80c690e61c0b4df48f823263c68ce68f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33151 } } peers { permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } }
I20260620 04:32:43.810546 13656 ts_tablet_manager.cc:1428] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 04:32:43.810878 13620 heartbeater.cc:503] Master 127.0.98.62:39775 was elected leader, sending a full tablet report...
I20260620 04:32:43.812825 13654 raft_consensus.cc:348] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 } } peers { permanent_uuid: "80c690e61c0b4df48f823263c68ce68f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33151 } } peers { permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } }
I20260620 04:32:43.813154 13654 raft_consensus.cc:374] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:43.813215 13654 raft_consensus.cc:729] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c690e61c0b4df48f823263c68ce68f, State: Initialized, Role: FOLLOWER
I20260620 04:32:43.813709 13654 consensus_queue.cc:260] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 } } peers { permanent_uuid: "80c690e61c0b4df48f823263c68ce68f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33151 } } peers { permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } }
I20260620 04:32:43.814867 13654 ts_tablet_manager.cc:1428] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:32:43.815265 13486 heartbeater.cc:503] Master 127.0.98.62:39775 was elected leader, sending a full tablet report...
W20260620 04:32:43.925181 13353 tablet.cc:2369] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:32:43.935242 13621 tablet.cc:2369] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:32:43.936281 13487 tablet.cc:2369] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:32:44.093564 13662 raft_consensus.cc:482] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:32:44.093809 13662 raft_consensus.cc:504] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 } } peers { permanent_uuid: "80c690e61c0b4df48f823263c68ce68f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33151 } } peers { permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } }
I20260620 04:32:44.095216 13662 leader_election.cc:288] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 087fcf524c4f4de09c1e653e1923806f (127.0.98.1:44963), d4665721111e4ef890426feeb623d62a (127.0.98.3:32951)
I20260620 04:32:44.102172 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:44.096523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49537 (local address 127.0.98.3:32951)
0620 04:32:44.096654 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:44.096658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:44.097596 (+ 938us) server_negotiation.cc:407] Connection header received
0620 04:32:44.097845 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:44.097848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:44.097901 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:44.097946 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:44.098353 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:44.099137 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:44.100851 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:44.101464 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:44.101556 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:44.101626 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:44.101951 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 04:32:44.102036 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:32:44.102665 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:44.096450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32951 (local address 127.0.98.2:49537)
0620 04:32:44.097479 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:44.097496 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:44.097723 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:44.098081 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:44.098084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:44.098099 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:44.098235 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:44.098240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:44.099302 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:44.099305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:44.100706 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:44.100718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:44.101723 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:44.101726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:44.101823 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:44.101845 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:32:44.102351 (+ 506us) client_negotiation.cc:240] Negotiation successful
0620 04:32:44.102433 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":99,"threads_started":1}
I20260620 04:32:44.103376 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:44.097323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44211 (local address 127.0.98.1:44963)
0620 04:32:44.097442 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:44.097447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:44.097695 (+ 248us) server_negotiation.cc:407] Connection header received
0620 04:32:44.097946 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:44.097950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:44.098002 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:44.098048 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:44.098438 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:44.099178 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:44.102056 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:44.102649 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:44.102732 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:44.102794 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:44.103184 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 04:32:44.103266 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:32:44.103504 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1720df409c5413fa347b41f08df1527" candidate_uuid: "80c690e61c0b4df48f823263c68ce68f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4665721111e4ef890426feeb623d62a" is_pre_election: true
I20260620 04:32:44.103894 13574 raft_consensus.cc:2454] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c690e61c0b4df48f823263c68ce68f in term 0.
I20260620 04:32:44.103942 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:44.096869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44963 (local address 127.0.98.2:44211)
0620 04:32:44.097608 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:44.097622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:44.097811 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:44.098179 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:44.098182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:44.098198 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:44.098315 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:44.098321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:44.099317 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:44.099320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:44.101909 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:44.101921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:44.102949 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:44.102953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:44.103043 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:44.103066 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:32:44.103596 (+ 530us) client_negotiation.cc:240] Negotiation successful
0620 04:32:44.103798 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":55,"thread_start_us":123,"threads_started":1}
I20260620 04:32:44.104470 13375 leader_election.cc:302] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c690e61c0b4df48f823263c68ce68f, d4665721111e4ef890426feeb623d62a; no voters:
I20260620 04:32:44.104589 13304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1720df409c5413fa347b41f08df1527" candidate_uuid: "80c690e61c0b4df48f823263c68ce68f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087fcf524c4f4de09c1e653e1923806f" is_pre_election: true
I20260620 04:32:44.104823 13662 raft_consensus.cc:2781] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:32:44.104915 13662 raft_consensus.cc:482] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:32:44.104974 13662 raft_consensus.cc:3037] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:44.104955 13304 raft_consensus.cc:2454] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c690e61c0b4df48f823263c68ce68f in term 0.
I20260620 04:32:44.106390 13662 raft_consensus.cc:504] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 } } peers { permanent_uuid: "80c690e61c0b4df48f823263c68ce68f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33151 } } peers { permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } }
I20260620 04:32:44.106817 13662 leader_election.cc:288] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [CANDIDATE]: Term 1 election: Requested vote from peers 087fcf524c4f4de09c1e653e1923806f (127.0.98.1:44963), d4665721111e4ef890426feeb623d62a (127.0.98.3:32951)
I20260620 04:32:44.107209 13304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1720df409c5413fa347b41f08df1527" candidate_uuid: "80c690e61c0b4df48f823263c68ce68f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087fcf524c4f4de09c1e653e1923806f"
I20260620 04:32:44.107364 13304 raft_consensus.cc:3037] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:44.107313 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1720df409c5413fa347b41f08df1527" candidate_uuid: "80c690e61c0b4df48f823263c68ce68f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4665721111e4ef890426feeb623d62a"
I20260620 04:32:44.107544 13574 raft_consensus.cc:3037] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:44.108743 13304 raft_consensus.cc:2454] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c690e61c0b4df48f823263c68ce68f in term 1.
I20260620 04:32:44.108877 13574 raft_consensus.cc:2454] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c690e61c0b4df48f823263c68ce68f in term 1.
I20260620 04:32:44.109258 13375 leader_election.cc:302] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c690e61c0b4df48f823263c68ce68f, d4665721111e4ef890426feeb623d62a; no voters:
I20260620 04:32:44.109551 13662 raft_consensus.cc:2781] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:32:44.109797 13662 raft_consensus.cc:686] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 1 LEADER]: Becoming Leader. State: Replica: 80c690e61c0b4df48f823263c68ce68f, State: Running, Role: LEADER
I20260620 04:32:44.110209 13662 consensus_queue.cc:237] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 } } peers { permanent_uuid: "80c690e61c0b4df48f823263c68ce68f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33151 } } peers { permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } }
I20260620 04:32:44.116254 13153 catalog_manager.cc:5495] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80c690e61c0b4df48f823263c68ce68f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "80c690e61c0b4df48f823263c68ce68f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80c690e61c0b4df48f823263c68ce68f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } health_report { overall_health: UNKNOWN } } }
I20260620 04:32:44.212956 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:44.208829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.0.98.2:33151)
0620 04:32:44.208968 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:44.208973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:44.208990 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:32:44.209161 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:44.209164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:44.209221 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:44.209310 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:32:44.209586 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:44.210209 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:44.211422 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:44.211652 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:44.211743 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:44.212099 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:32:44.212545 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:32:44.212646 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:44.212733 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 04:32:44.212790 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:32:44.219658 13304 raft_consensus.cc:1261] T f1720df409c5413fa347b41f08df1527 P 087fcf524c4f4de09c1e653e1923806f [term 1 FOLLOWER]: Refusing update from remote peer 80c690e61c0b4df48f823263c68ce68f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:32:44.219658 13574 raft_consensus.cc:1261] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a [term 1 FOLLOWER]: Refusing update from remote peer 80c690e61c0b4df48f823263c68ce68f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:32:44.220436 13662 consensus_queue.cc:1035] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4665721111e4ef890426feeb623d62a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:32:44.220664 13673 consensus_queue.cc:1035] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [LEADER]: Connected to new peer: Peer: permanent_uuid: "087fcf524c4f4de09c1e653e1923806f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:32:44.228366 13676 mvcc.cc:205] Tried to move back new op lower bound from 7298785133436018688 to 7298785133000138752. Current Snapshot: MvccSnapshot[applied={T|T < 7298785133436018688}]
I20260620 04:32:44.239441 13153 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:32:44.239612 13153 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:32:44.241914 13153 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63e44508bc7a4bf0aa7062acc45a42ed: Sending DeleteTablet for 3 replicas of tablet f1720df409c5413fa347b41f08df1527
I20260620 04:32:44.242684 13267 tablet_service.cc:1514] Processing DeleteTablet for tablet f1720df409c5413fa347b41f08df1527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:32:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43152
I20260620 04:32:44.242786 13420 tablet_service.cc:1514] Processing DeleteTablet for tablet f1720df409c5413fa347b41f08df1527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:32:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53132
I20260620 04:32:44.243263 13554 tablet_service.cc:1514] Processing DeleteTablet for tablet f1720df409c5413fa347b41f08df1527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:32:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50868
I20260620 04:32:44.243383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 13192
I20260620 04:32:44.249228 13686 tablet_replica.cc:330] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: stopping tablet replica
I20260620 04:32:44.249251 13688 tablet_replica.cc:330] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: stopping tablet replica
I20260620 04:32:44.249594 13686 raft_consensus.cc:2229] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:32:44.249677 13688 raft_consensus.cc:2229] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:32:44.250038 13686 raft_consensus.cc:2258] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:32:44.250072 13688 raft_consensus.cc:2258] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:32:44.251848 13688 ts_tablet_manager.cc:1905] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:32:44.251847 13686 ts_tablet_manager.cc:1905] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:32:44.254679 13686 ts_tablet_manager.cc:1918] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:32:44.254863 13686 log.cc:1196] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/f1720df409c5413fa347b41f08df1527
W20260620 04:32:44.255168 13138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44963 (error 108)
I20260620 04:32:44.255245 13686 ts_tablet_manager.cc:1939] T f1720df409c5413fa347b41f08df1527 P 80c690e61c0b4df48f823263c68ce68f: Deleting consensus metadata
I20260620 04:32:44.256665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 13354
I20260620 04:32:44.256740 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:44.256415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44963 (local address 127.0.0.1:43154)
0620 04:32:44.256559 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:44.256643 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 04:32:44.258776 13138 catalog_manager.cc:4584] TS 087fcf524c4f4de09c1e653e1923806f (127.0.98.1:44963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1720df409c5413fa347b41f08df1527: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44963: connect: Connection refused (error 111)
I20260620 04:32:44.260815 13688 ts_tablet_manager.cc:1918] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:32:44.260921 13688 log.cc:1196] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/f1720df409c5413fa347b41f08df1527
I20260620 04:32:44.261256 13688 ts_tablet_manager.cc:1939] T f1720df409c5413fa347b41f08df1527 P d4665721111e4ef890426feeb623d62a: Deleting consensus metadata
I20260620 04:32:44.262598 13140 catalog_manager.cc:4841] TS d4665721111e4ef890426feeb623d62a (127.0.98.3:32951): tablet f1720df409c5413fa347b41f08df1527 (table test-table [id=80dc62e6fbe646a29da6b41c7565fff7]) successfully deleted
W20260620 04:32:44.263358 13651 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:32:44.263052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33151 (local address 127.0.0.1:53148)
0620 04:32:44.263194 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:44.263252 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33151: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 04:32:44.263492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 13488
I20260620 04:32:44.269548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 13121
I20260620 04:32:44.276729 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 04:32:44.612056 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 04:32:44.612174 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 04:32:44.947078 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 04:32:44.947 UTC [12188] LOG: received smart shutdown request
2026-06-20 04:32:44.950 UTC [12188] LOG: background worker "logical replication launcher" (PID 12197) exited with exit code 1
2026-06-20 04:32:44.950 UTC [12192] LOG: shutting down
2026-06-20 04:32:45.117 UTC [12188] LOG: database system is shut down
2026-06-20T04:32:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47510 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:32:45 dist-test-slave-rzqm krb5kdc[13698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:32:45 dist-test-slave-rzqm krb5kdc[13698](info): set up 2 sockets
Jun 20 04:32:45 dist-test-slave-rzqm krb5kdc[13698](info): commencing operation
krb5kdc: starting...
W20260620 04:32:46.164820 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:32:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:32:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:32:46.289100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33555,127.0.98.61:35645,127.0.98.60:33339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:46.403046 13711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:46.403337 13711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:46.403378 13711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:46.407919 13711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:32:46.407997 13711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:46.408017 13711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:46.408033 13711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:32:46.408047 13711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:32:46.414075 13711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33555,127.0.98.61:35645,127.0.98.60:33339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:46.415239 13711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:46.416234 13711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:46.424484 13716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:46.424486 13717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:46.424538 13711 server_base.cc:1034] running on GCE node
W20260620 04:32:46.424533 13719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:46.425429 13711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:46.426498 13711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:46.427670 13711 hybrid_clock.cc:648] HybridClock initialized: now 1781929966427648 us; error 37 us; skew 500 ppm
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:46.430429 13711 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:32:46.431313 13711 webserver.cc:460] Webserver started at http://127.0.98.62:43739/ using document root <none> and password file <none>
I20260620 04:32:46.431855 13711 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:46.431916 13711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:46.432083 13711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:32:46.434173 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "22f8204ca0e34afdabea42315bf6c71e"
format_stamp: "Formatted at 2026-06-20 04:32:46 on dist-test-slave-rzqm"
server_key: "d52d84318021fcc827b6a5f4a24c75b0"
server_key_iv: "d880c1c8861f39faa7c72f919b87282b"
server_key_version: "encryptionkey@0"
I20260620 04:32:46.434708 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "22f8204ca0e34afdabea42315bf6c71e"
format_stamp: "Formatted at 2026-06-20 04:32:46 on dist-test-slave-rzqm"
server_key: "d52d84318021fcc827b6a5f4a24c75b0"
server_key_iv: "d880c1c8861f39faa7c72f919b87282b"
server_key_version: "encryptionkey@0"
I20260620 04:32:46.438433 13711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:32:46.440976 13726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:46.442541 13711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:32:46.442657 13711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "22f8204ca0e34afdabea42315bf6c71e"
format_stamp: "Formatted at 2026-06-20 04:32:46 on dist-test-slave-rzqm"
server_key: "d52d84318021fcc827b6a5f4a24c75b0"
server_key_iv: "d880c1c8861f39faa7c72f919b87282b"
server_key_version: "encryptionkey@0"
I20260620 04:32:46.442754 13711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:46.455039 13711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:46.458352 13711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:46.458626 13711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:46.465830 13711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33555
I20260620 04:32:46.465883 13778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33555 every 8 connection(s)
I20260620 04:32:46.466785 13711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:32:46.469588 13779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:46.473304 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:32:46.475972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 13711
I20260620 04:32:46.476084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:32:46.476359 392 external_mini_cluster.cc:1428] Setting key ff07ae1baa0bd6e20d9c8fde88665f9a
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 04:32:46.487982 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.475535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.0.98.62:33555)
0620 04:32:46.476040 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.476058 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.476269 (+ 211us) server_negotiation.cc:407] Connection header received
0620 04:32:46.477316 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.477320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.477512 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.477674 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.478527 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.479392 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.480348 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.480721 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.480871 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.483334 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.483340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.483351 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.483382 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.484936 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.485450 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.485456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.485457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.485595 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.485894 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.485897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.485899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.486386 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.486515 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.486900 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.487211 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":405,"spinlock_wait_cycles":42752,"thread_start_us":164,"threads_started":1}
I20260620 04:32:46.487986 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.475534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33555 (local address 127.0.0.1:56622)
0620 04:32:46.476061 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.476110 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:46.476984 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:46.477689 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:46.477694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:46.478136 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:46.478381 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.478394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.479506 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.479509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.480234 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.480243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.480874 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.480878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.480986 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.481585 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:46.481603 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:46.483158 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:46.485095 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.485104 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.485114 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.485354 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.485679 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.485682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.485684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.485809 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.486514 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:46.486520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:46.486655 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 04:32:46.486809 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 04:32:46.487097 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":136,"threads_started":1}
I20260620 04:32:46.490556 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:32:46.491286 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.491040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35645 (local address 127.0.0.1:43186)
0620 04:32:46.491158 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.491227 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 04:32:46.491530 13730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35645: connect: Connection refused (error 111)
I20260620 04:32:46.492425 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.492149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35645 (local address 127.0.0.1:43202)
0620 04:32:46.492282 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.492372 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 04:32:46.492584 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35645: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35645: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 04:32:46.500208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35645
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33555,127.0.98.61:35645,127.0.98.60:33339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:32:46.541831 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } attempt: 1
I20260620 04:32:46.542716 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.542408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35645 (local address 127.0.0.1:43218)
0620 04:32:46.542572 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.542637 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 04:32:46.543394 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.543180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35645 (local address 127.0.0.1:43232)
0620 04:32:46.543273 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.543331 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 04:32:46.543514 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35645: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35645: connect: Connection refused (error 111)
I20260620 04:32:46.611706 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } attempt: 2
I20260620 04:32:46.612654 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.612280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35645 (local address 127.0.0.1:43244)
0620 04:32:46.612509 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.612576 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260620 04:32:46.612665 13786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:46.612977 13786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:46.613096 13786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 04:32:46.613384 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.613175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35645 (local address 127.0.0.1:43260)
0620 04:32:46.613279 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.613324 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 04:32:46.613499 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35645: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35645: connect: Connection refused (error 111)
W20260620 04:32:46.617436 13786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:32:46.617496 13786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:46.617522 13786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:46.617555 13786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:32:46.617583 13786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:32:46.623066 13786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33555,127.0.98.61:35645,127.0.98.60:33339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:46.624166 13786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:46.625249 13786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:46.633488 13791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:46.633481 13792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:46.633505 13794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:46.633661 13786 server_base.cc:1034] running on GCE node
I20260620 04:32:46.634344 13786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:46.635509 13786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:46.636693 13786 hybrid_clock.cc:648] HybridClock initialized: now 1781929966636667 us; error 39 us; skew 500 ppm
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:46.639462 13786 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 04:32:46.640395 13786 webserver.cc:460] Webserver started at http://127.0.98.61:37619/ using document root <none> and password file <none>
I20260620 04:32:46.640909 13786 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:46.640966 13786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:46.641187 13786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:32:46.643190 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/data/instance:
uuid: "6fc0e392ff1c42498c58e9f1f554c4e0"
format_stamp: "Formatted at 2026-06-20 04:32:46 on dist-test-slave-rzqm"
server_key: "6a95b84316cbba1ebe83f34a5b75c354"
server_key_iv: "f9632c2302c5c7057d27fa174a4e644b"
server_key_version: "encryptionkey@0"
I20260620 04:32:46.643719 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/wal/instance:
uuid: "6fc0e392ff1c42498c58e9f1f554c4e0"
format_stamp: "Formatted at 2026-06-20 04:32:46 on dist-test-slave-rzqm"
server_key: "6a95b84316cbba1ebe83f34a5b75c354"
server_key_iv: "f9632c2302c5c7057d27fa174a4e644b"
server_key_version: "encryptionkey@0"
I20260620 04:32:46.647411 13786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:32:46.649852 13801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:46.651214 13786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:32:46.651325 13786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/wal
uuid: "6fc0e392ff1c42498c58e9f1f554c4e0"
format_stamp: "Formatted at 2026-06-20 04:32:46 on dist-test-slave-rzqm"
server_key: "6a95b84316cbba1ebe83f34a5b75c354"
server_key_iv: "f9632c2302c5c7057d27fa174a4e644b"
server_key_version: "encryptionkey@0"
I20260620 04:32:46.651437 13786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:46.669407 13786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:46.672518 13786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:46.672712 13786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:46.679629 13786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35645
I20260620 04:32:46.679644 13853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35645 every 8 connection(s)
I20260620 04:32:46.680576 13786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:32:46.683369 13854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:46.686514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 13786
I20260620 04:32:46.686621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-1/wal/instance
I20260620 04:32:46.686878 392 external_mini_cluster.cc:1428] Setting key 40bf92693ce1903494a9d960715fe97e
I20260620 04:32:46.687216 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:46.700938 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.689225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.62:33555)
0620 04:32:46.689357 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.689360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.690486 (+ 1126us) server_negotiation.cc:407] Connection header received
0620 04:32:46.691452 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.691456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.691521 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.691589 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.692789 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.693359 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.694527 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.694761 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.694899 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.697524 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.697527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.697529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.697556 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.698890 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.699395 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.699398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.699399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.699477 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.699794 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.699797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.699799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.699986 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.700071 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.700613 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.700753 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:32:46.701817 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.689729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33555 (local address 127.0.0.1:56638)
0620 04:32:46.690254 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.690361 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:46.691229 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:46.691770 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:46.691777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:46.692145 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:46.692642 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.692655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.693480 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.693483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.694414 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.694424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.694950 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.694954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.695121 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.695689 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:46.695706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:46.697348 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:46.699005 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.699012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.699031 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.699292 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.699560 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.699563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.699565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.699708 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.700083 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:46.700091 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:46.700365 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 04:32:46.700543 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 04:32:46.700928 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":145,"threads_started":1}
I20260620 04:32:46.703266 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:32:46.704648 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } attempt: 3
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 04:32:46.711915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33339
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33555,127.0.98.61:35645,127.0.98.60:33339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:32:46.714524 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.703843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35645 (local address 127.0.0.1:43266)
0620 04:32:46.703985 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.703999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:46.704085 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:46.704752 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:46.704755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:46.704925 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:46.705070 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.705078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.706426 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.706429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.707371 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.707379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.708192 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.708195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.708271 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.708808 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:46.708822 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:46.710151 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:46.712941 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.712944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.712946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.713223 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.713494 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.713497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.713498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.713572 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.714060 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:46.714063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:46.714141 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 04:32:46.714221 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 04:32:46.714353 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"spinlock_wait_cycles":896}
I20260620 04:32:46.714581 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.705087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35645 (local address 127.0.0.1:43278)
0620 04:32:46.705196 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.705210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:46.705273 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:46.705716 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:46.705719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:46.705899 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:46.706420 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.706430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.707481 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.707484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.708269 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.708280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.708800 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.708804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.708902 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.709330 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:46.709345 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:46.710681 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:46.712871 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.712874 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.712876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.713122 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.713441 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.713445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.713447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.713537 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.714096 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:46.714099 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:46.714200 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 04:32:46.714282 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 04:32:46.714421 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 04:32:46.714630 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.703970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.0.98.61:35645)
0620 04:32:46.704372 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.704381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.704400 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:32:46.704433 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.704438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.704602 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.704744 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.705192 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.706301 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.707480 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.708040 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.708147 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.710297 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.710306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.710360 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.710381 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.712834 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.713333 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.713335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.713337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.713415 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.713656 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.713658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.713660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.713966 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.714082 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.714334 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.714462 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":312,"thread_start_us":163,"threads_started":1}
I20260620 04:32:46.715194 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.705139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.0.98.61:35645)
0620 04:32:46.705465 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.705474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.705487 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:32:46.705518 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.705521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.705561 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.705675 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.706523 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.707340 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.708382 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.708680 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.708785 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.710867 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.710871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.710874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.710898 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.712767 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.713231 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.713238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.713240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.713331 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.713636 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.713639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.713640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.713982 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.714865 (+ 883us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.714939 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.715039 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":263,"thread_start_us":128,"threads_started":1}
I20260620 04:32:46.716562 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:32:46.716651 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:32:46.717310 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.717048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:57970)
0620 04:32:46.717173 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.717235 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 04:32:46.717350 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.717095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:57976)
0620 04:32:46.717229 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.717293 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 04:32:46.717662 13804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
I20260620 04:32:46.718001 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.717728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:57982)
0620 04:32:46.717886 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.717938 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 04:32:46.718132 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
I20260620 04:32:46.718617 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.718438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:57992)
0620 04:32:46.718527 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.718571 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 04:32:46.718725 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
I20260620 04:32:46.749292 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } attempt: 1
I20260620 04:32:46.750183 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.749851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58006)
0620 04:32:46.750026 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.750095 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 04:32:46.750924 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.750720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58008)
0620 04:32:46.750826 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.750869 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 04:32:46.751070 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
I20260620 04:32:46.767954 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } attempt: 1
I20260620 04:32:46.768796 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.768517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58010)
0620 04:32:46.768658 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.768719 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 04:32:46.769497 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.769294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58020)
0620 04:32:46.769399 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.769440 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 04:32:46.769601 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
W20260620 04:32:46.825196 13861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:46.825614 13861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:46.825706 13861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 04:32:46.826224 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } attempt: 2
I20260620 04:32:46.827112 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.826686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58026)
0620 04:32:46.826961 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.827032 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260620 04:32:46.827893 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.827676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58032)
0620 04:32:46.827780 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.827826 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 04:32:46.828054 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
W20260620 04:32:46.830154 13861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:32:46.830207 13861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:46.830235 13861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:46.830268 13861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:32:46.830320 13861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:32:46.835767 13861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33555,127.0.98.61:35645,127.0.98.60:33339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:46.836843 13861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:46.837741 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } attempt: 2
I20260620 04:32:46.837877 13861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:32:46.838465 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.838200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58036)
0620 04:32:46.838335 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.838394 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 04:32:46.839239 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.839026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58050)
0620 04:32:46.839140 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.839184 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 04:32:46.839342 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33339: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33339: connect: Connection refused (error 111)
W20260620 04:32:46.846659 13868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:46.846674 13867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:46.846674 13870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:46.846854 13861 server_base.cc:1034] running on GCE node
I20260620 04:32:46.847533 13861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:46.848625 13861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:46.849785 13861 hybrid_clock.cc:648] HybridClock initialized: now 1781929966849761 us; error 37 us; skew 500 ppm
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:46.852502 13861 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 04:32:46.853427 13861 webserver.cc:460] Webserver started at http://127.0.98.60:39517/ using document root <none> and password file <none>
I20260620 04:32:46.853965 13861 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:46.854033 13861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:46.854228 13861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:32:46.856216 13861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/data/instance:
uuid: "880437e321414519a09004355d380d5c"
format_stamp: "Formatted at 2026-06-20 04:32:46 on dist-test-slave-rzqm"
server_key: "43f7c84577bc1c089bcffff2427cf751"
server_key_iv: "25e4aeacf23e7bc03edac522af28aa44"
server_key_version: "encryptionkey@0"
I20260620 04:32:46.856739 13861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/wal/instance:
uuid: "880437e321414519a09004355d380d5c"
format_stamp: "Formatted at 2026-06-20 04:32:46 on dist-test-slave-rzqm"
server_key: "43f7c84577bc1c089bcffff2427cf751"
server_key_iv: "25e4aeacf23e7bc03edac522af28aa44"
server_key_version: "encryptionkey@0"
I20260620 04:32:46.860349 13861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:32:46.862748 13877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:46.864074 13861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:32:46.864207 13861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/wal
uuid: "880437e321414519a09004355d380d5c"
format_stamp: "Formatted at 2026-06-20 04:32:46 on dist-test-slave-rzqm"
server_key: "43f7c84577bc1c089bcffff2427cf751"
server_key_iv: "25e4aeacf23e7bc03edac522af28aa44"
server_key_version: "encryptionkey@0"
I20260620 04:32:46.864308 13861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:46.896863 13861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:46.900126 13861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:46.900311 13861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:46.907123 13861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33339
I20260620 04:32:46.907181 13929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33339 every 8 connection(s)
I20260620 04:32:46.908092 13861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:32:46.909384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 13861
I20260620 04:32:46.909510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781929807654528-392-0/minicluster-data/master-2/wal/instance
I20260620 04:32:46.909796 392 external_mini_cluster.cc:1428] Setting key 69dde26f5d963622b1e5d5d86856dd7b
I20260620 04:32:46.911499 13930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:46.915635 13930 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:46.924638 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.911484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.0.98.60:33339)
0620 04:32:46.911880 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.911899 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.911940 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:32:46.912642 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.912660 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.913023 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.913303 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.914191 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.915199 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.916320 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.916791 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.916910 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.919717 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.919723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.919732 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.919760 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.921642 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.922068 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.922075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.922081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.922206 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.922474 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.922477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.922478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.922893 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.923123 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.923369 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.923674 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":126,"threads_started":1}
I20260620 04:32:46.926441 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.917523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.0.98.62:33555)
0620 04:32:46.917649 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.917653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.918027 (+ 374us) server_negotiation.cc:407] Connection header received
0620 04:32:46.918144 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.918147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.918203 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.918290 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.918988 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.919708 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.920826 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.921089 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.921170 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.923356 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.923359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.923361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.923385 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.924709 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.925227 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.925230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.925233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.925312 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.925571 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.925575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.925576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.925756 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.925840 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.926181 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.926290 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:32:46.926723 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.917487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33555 (local address 127.0.0.1:56642)
0620 04:32:46.917888 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.917936 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:46.918056 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:46.918351 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:46.918355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:46.918734 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:46.918856 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.918861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.919826 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.919829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.920714 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.920725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.921194 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.921198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.921313 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.921781 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:46.921794 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:46.923198 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:46.924835 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.924844 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.924850 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.925133 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.925396 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.925399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.925401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.925476 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.925855 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:46.925861 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:46.925969 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 04:32:46.926096 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 04:32:46.926282 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":135,"threads_started":1}
I20260620 04:32:46.927191 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } attempt: 3
I20260620 04:32:46.928087 13930 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:32:46.930485 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } attempt: 3
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:32:46.937331 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.926927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.0.98.62:33555)
0620 04:32:46.927029 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.927033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.927096 (+ 63us) server_negotiation.cc:407] Connection header received
0620 04:32:46.927214 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.927217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.927260 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.927335 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.928330 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.929112 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.930171 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.930564 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.930654 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.933611 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.933615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.933618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.933644 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.935162 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.935529 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.935533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.935535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.935618 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.935891 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.935895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.935897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.936113 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.936338 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.937044 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.937165 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":40}
I20260620 04:32:46.938177 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.927531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58056)
0620 04:32:46.927630 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.927643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:46.927706 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:46.928252 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:46.928255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:46.928418 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:46.928506 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.928512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.929416 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.929419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.930235 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.930244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.931272 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.931276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.931365 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.932070 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:46.932085 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:46.934171 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:46.936697 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.936700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.936703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.936919 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.937255 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.937258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.937260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.937333 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.937734 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:46.937738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:46.937837 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 04:32:46.937903 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 04:32:46.938036 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":23}
I20260620 04:32:46.938530 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.928627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.0.98.61:35645)
0620 04:32:46.928737 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.928741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.928754 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:32:46.928819 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.928822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.928865 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.928927 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.929393 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.929906 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.930911 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.931754 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.931838 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.934595 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.934599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.934608 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.934641 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.936416 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.936873 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.936876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.936878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.936966 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.937300 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.937303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.937305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.937518 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.937585 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.938259 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.938388 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:32:46.938650 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.927604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.0.98.60:33339)
0620 04:32:46.928069 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.928073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.928084 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:32:46.928114 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.928116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.928153 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.928218 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.928628 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.929333 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.930338 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.931145 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.931568 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.934938 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.934942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.934944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.934991 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.936595 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.937074 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.937076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.937078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.937157 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.937438 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.937441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.937442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.937632 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.938329 (+ 697us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.938402 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.938510 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"spinlock_wait_cycles":4864}
I20260620 04:32:46.939648 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.928518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35645 (local address 127.0.0.1:43290)
0620 04:32:46.928660 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.928673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:46.928738 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:46.928952 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:46.928955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:46.929162 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:46.929265 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.929272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.930020 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.930023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.930782 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.930791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.931994 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.931997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.932098 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.932522 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:46.932537 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:46.934438 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:46.936538 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.936541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.936543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.936768 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.937111 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.937113 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.937115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.937197 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.937960 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:46.937963 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:46.938056 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 04:32:46.939386 (+ 1330us) client_negotiation.cc:240] Negotiation successful
0620 04:32:46.939509 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 04:32:46.940476 13930 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:32:46.941093 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.930892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58072)
0620 04:32:46.931174 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.931188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:46.931255 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:46.931696 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:46.931699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:46.931855 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:46.931928 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.931934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.932887 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.932891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.933553 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.933565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.934758 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.934762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.934847 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.935357 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:46.935374 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:46.936826 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:46.939457 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.939460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.939462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.939681 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.939974 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.939977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.939979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.940062 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.940637 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:46.940640 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:46.940728 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 04:32:46.940805 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 04:32:46.940925 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260620 04:32:46.941862 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.931149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.0.98.60:33339)
0620 04:32:46.931503 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.931512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.931524 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:32:46.931555 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.931557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.931595 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.931673 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.932043 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.932779 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.934289 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.934648 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.934738 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.936949 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.936953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.936955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.936976 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.939340 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.939799 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.939802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.939804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.939877 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.940164 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.940168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.940170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.940532 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.941567 (+ 1035us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.941624 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.941714 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":148,"threads_started":1}
I20260620 04:32:46.942056 13779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e: Bootstrap starting.
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:32:46.944959 13779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:46.944944 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0: Bootstrap starting.
I20260620 04:32:46.945912 13779 log.cc:824] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:46.947444 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.938723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.0.98.61:35645)
0620 04:32:46.939151 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.939155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.939169 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:32:46.939201 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.939204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.939244 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.939316 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.940025 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.940718 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.941757 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.942072 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.942167 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.944346 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.944349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.944352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.944376 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.945625 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.946010 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.946013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.946015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.946091 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.946359 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.946362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.946363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.946572 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.946750 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.947112 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.947236 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":355}
I20260620 04:32:46.947721 13854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:46.948544 13854 log.cc:824] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:46.951040 13779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e: No bootstrap required, opened a new log
I20260620 04:32:46.951040 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0: No bootstrap required, opened a new log
Jun 20 04:32:46 dist-test-slave-rzqm krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:32:46.951551 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:32:46.954749 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.940934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33339 (local address 127.0.0.1:58078)
0620 04:32:46.941264 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:46.941277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:46.941341 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:46.946819 (+ 5478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:46.946822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:46.947093 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:46.947213 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.947222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.948091 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.948094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.948752 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:46.948760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.949355 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.949359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:46.949437 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.949929 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:46.949943 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:46.951440 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:46.953287 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.953290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.953293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.953519 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.953786 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:46.953789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:46.953791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:46.953862 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:46.954249 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:46.954252 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:46.954324 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 04:32:46.954403 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 04:32:46.954544 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260620 04:32:46.955330 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.946454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58078 (local address 127.0.98.60:33339)
0620 04:32:46.946567 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.946571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.946585 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:32:46.946619 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.946622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.946667 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.947314 (+ 647us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.947387 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.947990 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.948861 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.949232 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.949661 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.951582 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.951586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.951589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.951612 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.953175 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.953619 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.953622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.953624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.953690 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.953943 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.953946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.953948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.954160 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.954926 (+ 766us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.955010 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.955190 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 04:32:46.955263 13779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } }
I20260620 04:32:46.955263 13854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } }
I20260620 04:32:46.955510 13854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:46.955533 13779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:46.955585 13854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc0e392ff1c42498c58e9f1f554c4e0, State: Initialized, Role: FOLLOWER
I20260620 04:32:46.955617 13779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f8204ca0e34afdabea42315bf6c71e, State: Initialized, Role: FOLLOWER
I20260620 04:32:46.956244 13779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } }
I20260620 04:32:46.956347 13854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } }
I20260620 04:32:46.956791 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } } }
I20260620 04:32:46.957033 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:32:46.957650 13854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:32:46.957697 13779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:32:46.958666 13930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c: Bootstrap starting.
W20260620 04:32:46.960227 13732 tablet.cc:2369] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:32:46.961977 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } } }
I20260620 04:32:46.962146 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:32:46.962255 13930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:46.963130 13930 log.cc:824] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c: Log is configured to *not* fsync() on all Append() calls
W20260620 04:32:46.963171 13968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:32:46.963238 13968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:32:46.970032 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.954609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.0.98.60:33339)
0620 04:32:46.954737 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.954742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.960589 (+ 5847us) server_negotiation.cc:407] Connection header received
0620 04:32:46.960792 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.960795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.960842 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.960892 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.962271 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.963011 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.964053 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.964355 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.964447 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.966546 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.966550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.966553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.966580 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.968239 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.968674 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.968678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.968680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.968770 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.969099 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.969102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.969104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.969326 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.969431 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.969752 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.969866 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 04:32:46.970125 13945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:32:46.970276 13945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } }
I20260620 04:32:46.971830 13945 leader_election.cc:288] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22f8204ca0e34afdabea42315bf6c71e (127.0.98.62:33555), 880437e321414519a09004355d380d5c (127.0.98.60:33339)
I20260620 04:32:46.972821 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.955870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.0.98.62:33555)
0620 04:32:46.955991 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.955995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.964292 (+ 8297us) server_negotiation.cc:407] Connection header received
0620 04:32:46.964804 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.964808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.964854 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.964898 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.965821 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.966557 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.967607 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.967941 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.968021 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.969584 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.969588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.969591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.969615 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.971122 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.971569 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.971573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.971575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.971658 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.971932 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.971935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.971937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.972137 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.972240 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.972563 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.972677 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 04:32:46.973816 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:32:46.973891 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:32:46.977478 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22f8204ca0e34afdabea42315bf6c71e" is_pre_election: true
I20260620 04:32:46.977689 13930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c: No bootstrap required, opened a new log
I20260620 04:32:46.977823 13753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fc0e392ff1c42498c58e9f1f554c4e0 in term 0.
I20260620 04:32:46.981858 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:46.964078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.0.98.61:35645)
0620 04:32:46.965314 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:46.965318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:46.965331 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:32:46.965363 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:46.965366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:46.965410 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:46.965482 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:46.974033 (+ 8551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.974709 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.975504 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:46.975789 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:46.975891 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:46.977659 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:46.977663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:46.977665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:46.977694 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:46.979645 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.980127 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.980130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.980132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.980217 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:46.980500 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:46.980503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:46.980505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:46.980725 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:46.980840 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:46.981598 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 04:32:46.981711 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162}
I20260620 04:32:46.981870 13930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } }
I20260620 04:32:46.982184 13930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:46.982251 13930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880437e321414519a09004355d380d5c, State: Initialized, Role: FOLLOWER
I20260620 04:32:46.979424 13786 master_runner.cc:186] Error getting master registration for 127.0.98.60:33339: OK, instance_id { permanent_uuid: "880437e321414519a09004355d380d5c" instance_seqno: 1781929966905146 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:32:46.982787 13930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } }
I20260620 04:32:46.983352 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } } }
I20260620 04:32:46.983611 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:32:46.984244 13930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:32:46.984282 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880437e321414519a09004355d380d5c" is_pre_election: true
I20260620 04:32:46.984545 13904 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fc0e392ff1c42498c58e9f1f554c4e0 in term 0.
I20260620 04:32:46.983738 13802 leader_election.cc:302] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22f8204ca0e34afdabea42315bf6c71e, 6fc0e392ff1c42498c58e9f1f554c4e0; no voters:
I20260620 04:32:46.985085 13786 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:32:46.985255 13945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:32:46.985328 13945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:32:46.985368 13945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:46.986635 13945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } }
I20260620 04:32:46.987403 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22f8204ca0e34afdabea42315bf6c71e"
I20260620 04:32:46.987542 13753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:46.988822 13753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fc0e392ff1c42498c58e9f1f554c4e0 in term 1.
I20260620 04:32:46.990718 13802 leader_election.cc:302] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22f8204ca0e34afdabea42315bf6c71e, 6fc0e392ff1c42498c58e9f1f554c4e0; no voters:
W20260620 04:32:46.990731 13984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:32:46.990808 13984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:32:46.991077 13986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:32:46.991318 13986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [term 1 LEADER]: Becoming Leader. State: Replica: 6fc0e392ff1c42498c58e9f1f554c4e0, State: Running, Role: LEADER
I20260620 04:32:46.991715 13986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } }
I20260620 04:32:46.994318 13945 leader_election.cc:288] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [CANDIDATE]: Term 1 election: Requested vote from peers 22f8204ca0e34afdabea42315bf6c71e (127.0.98.62:33555), 880437e321414519a09004355d380d5c (127.0.98.60:33339)
I20260620 04:32:46.994442 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc0e392ff1c42498c58e9f1f554c4e0. Latest consensus state: current_term: 1 leader_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } } }
I20260620 04:32:46.994550 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:46.995149 13989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:32:46.996198 13989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:32:46.996191 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880437e321414519a09004355d380d5c"
I20260620 04:32:46.996315 13904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:46.997668 13904 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fc0e392ff1c42498c58e9f1f554c4e0 in term 1.
I20260620 04:32:46.973390 13711 master_runner.cc:186] Error getting master registration for 127.0.98.60:33339: OK, instance_id { permanent_uuid: "880437e321414519a09004355d380d5c" instance_seqno: 1781929966905146 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:32:47.000419 13711 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:32:47.025811 13904 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [term 1 FOLLOWER]: Refusing update from remote peer 6fc0e392ff1c42498c58e9f1f554c4e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:32:47.026736 13945 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:32:47.027182 13753 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [term 1 FOLLOWER]: Refusing update from remote peer 6fc0e392ff1c42498c58e9f1f554c4e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:32:47.031260 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc0e392ff1c42498c58e9f1f554c4e0. Latest consensus state: current_term: 1 leader_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } } }
I20260620 04:32:47.031400 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:32:47.031783 13945 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:32:47.037663 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc0e392ff1c42498c58e9f1f554c4e0. Latest consensus state: current_term: 1 leader_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } } }
I20260620 04:32:47.037804 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:32:47.043860 13986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } } }
I20260620 04:32:47.044006 13986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:47.044145 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } } }
I20260620 04:32:47.044261 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:32:47.045681 13991 mvcc.cc:205] Tried to move back new op lower bound from 7298785144905650176 to 7298785144801693696. Current Snapshot: MvccSnapshot[applied={T|T < 7298785144905650176}]
I20260620 04:32:47.046113 13986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } } }
I20260620 04:32:47.046233 13986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0 [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:47.047346 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f8204ca0e34afdabea42315bf6c71e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "6fc0e392ff1c42498c58e9f1f554c4e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35645 } } peers { permanent_uuid: "880437e321414519a09004355d380d5c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33339 } } }
I20260620 04:32:47.047485 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:32:47.058828 13989 catalog_manager.cc:1347] Generated new cluster ID: 3b8b5beee78b4a1cb0d81ae6c7e1fab0
I20260620 04:32:47.058912 13989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:32:47.073051 13989 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:32:47.074481 13989 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:32:47.085474 13989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fc0e392ff1c42498c58e9f1f554c4e0: Generated new TSK 0
I20260620 04:32:47.086632 13989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:32:47.975214 13967 catalog_manager.cc:1259] Loaded cluster ID: 3b8b5beee78b4a1cb0d81ae6c7e1fab0
I20260620 04:32:47.975415 13967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e: loading cluster ID for follower catalog manager: success
I20260620 04:32:47.977363 13967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e: acquiring CA information for follower catalog manager: success
I20260620 04:32:47.978696 13967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22f8204ca0e34afdabea42315bf6c71e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:32:47.992115 13984 catalog_manager.cc:1259] Loaded cluster ID: 3b8b5beee78b4a1cb0d81ae6c7e1fab0
I20260620 04:32:47.992206 13984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c: loading cluster ID for follower catalog manager: success
I20260620 04:32:47.994076 13984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c: acquiring CA information for follower catalog manager: success
I20260620 04:32:47.995332 13984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 880437e321414519a09004355d380d5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:32:48.182178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 13711
I20260620 04:32:48.187934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 13786
I20260620 04:32:48.193068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 13861
2026-06-20T04:32:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2984 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:32:48 dist-test-slave-rzqm krb5kdc[14003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:32:48 dist-test-slave-rzqm krb5kdc[14003](info): set up 2 sockets
Jun 20 04:32:48 dist-test-slave-rzqm krb5kdc[14003](info): commencing operation
krb5kdc: starting...
W20260620 04:32:49.160983 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:32:49 dist-test-slave-rzqm krb5kdc[14003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:32:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:32:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:32:49.279718 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:49.390220 14016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:49.390501 14016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:49.390540 14016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:49.394616 14016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:32:49.394691 14016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:49.394709 14016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:49.394726 14016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:32:49.394743 14016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:32:49.400339 14016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:49.401481 14016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:49.402468 14016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:49.410938 14024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:49.411079 14021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:49.411132 14022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:49.411280 14016 server_base.cc:1034] running on GCE node
I20260620 04:32:49.411893 14016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:49.413041 14016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:49.414232 14016 hybrid_clock.cc:648] HybridClock initialized: now 1781929969414203 us; error 44 us; skew 500 ppm
Jun 20 04:32:49 dist-test-slave-rzqm krb5kdc[14003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:49.416908 14016 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:32:49.417810 14016 webserver.cc:460] Webserver started at http://127.0.98.62:36679/ using document root <none> and password file <none>
I20260620 04:32:49.418321 14016 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:49.418376 14016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:49.418588 14016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:32:49.420652 14016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "fd91c2c0fa8b4803b0836d38f119ace9"
format_stamp: "Formatted at 2026-06-20 04:32:49 on dist-test-slave-rzqm"
server_key: "f288912a381398c8dc6f148e18ee9223"
server_key_iv: "fa592c104c81b265c58785e332714ab4"
server_key_version: "encryptionkey@0"
I20260620 04:32:49.421233 14016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd91c2c0fa8b4803b0836d38f119ace9"
format_stamp: "Formatted at 2026-06-20 04:32:49 on dist-test-slave-rzqm"
server_key: "f288912a381398c8dc6f148e18ee9223"
server_key_iv: "fa592c104c81b265c58785e332714ab4"
server_key_version: "encryptionkey@0"
I20260620 04:32:49.425097 14016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:32:49.427830 14031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:49.429322 14016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:32:49.429464 14016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "fd91c2c0fa8b4803b0836d38f119ace9"
format_stamp: "Formatted at 2026-06-20 04:32:49 on dist-test-slave-rzqm"
server_key: "f288912a381398c8dc6f148e18ee9223"
server_key_iv: "fa592c104c81b265c58785e332714ab4"
server_key_version: "encryptionkey@0"
I20260620 04:32:49.429587 14016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:49.450278 14016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:49.453974 14016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:49.454229 14016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:49.461246 14016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35297
I20260620 04:32:49.461277 14083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35297 every 8 connection(s)
I20260620 04:32:49.462224 14016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:32:49.465046 14084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:49.466100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 14016
I20260620 04:32:49.466219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:32:49.466490 392 external_mini_cluster.cc:1428] Setting key d8a2bb001239b2e2f6453ea432c4b809
I20260620 04:32:49.472082 14084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9: Bootstrap starting.
I20260620 04:32:49.474448 14084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9: Neither blocks nor log segments found. Creating new log.
Jun 20 04:32:49 dist-test-slave-rzqm krb5kdc[14003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:49.475194 14084 log.cc:824] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:49.477310 14084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9: No bootstrap required, opened a new log
I20260620 04:32:49.480046 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:49.467719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.0.98.62:35297)
0620 04:32:49.468119 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:49.468135 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:49.468173 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:32:49.468880 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:49.468900 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:49.469297 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:49.469612 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:49.470431 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:49.471440 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:49.472331 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:49.472820 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:49.472980 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:49.475521 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:49.475527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:49.475542 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:49.475573 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:49.477223 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:49.477688 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:49.477693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:49.477698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:49.477817 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:49.478104 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:49.478107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:49.478108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:49.478538 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:49.478717 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:49.479013 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 04:32:49.479304 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260620 04:32:49.481775 14084 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd91c2c0fa8b4803b0836d38f119ace9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35297 } }
I20260620 04:32:49.482056 14084 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:49.482123 14084 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd91c2c0fa8b4803b0836d38f119ace9, State: Initialized, Role: FOLLOWER
I20260620 04:32:49.482583 14084 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd91c2c0fa8b4803b0836d38f119ace9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35297 } }
I20260620 04:32:49.482717 14084 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:32:49.482776 14084 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:32:49.482870 14084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:49.483098 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:32:49.483857 14084 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd91c2c0fa8b4803b0836d38f119ace9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35297 } }
I20260620 04:32:49.484196 14084 leader_election.cc:302] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd91c2c0fa8b4803b0836d38f119ace9; no voters:
I20260620 04:32:49.484550 14084 leader_election.cc:288] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:32:49.484637 14089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:32:49.484802 14089 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [term 1 LEADER]: Becoming Leader. State: Replica: fd91c2c0fa8b4803b0836d38f119ace9, State: Running, Role: LEADER
I20260620 04:32:49.485144 14089 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd91c2c0fa8b4803b0836d38f119ace9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35297 } }
I20260620 04:32:49.485908 14084 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:32:49.487007 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd91c2c0fa8b4803b0836d38f119ace9. Latest consensus state: current_term: 1 leader_uuid: "fd91c2c0fa8b4803b0836d38f119ace9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd91c2c0fa8b4803b0836d38f119ace9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35297 } } }
I20260620 04:32:49.487025 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd91c2c0fa8b4803b0836d38f119ace9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd91c2c0fa8b4803b0836d38f119ace9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35297 } } }
I20260620 04:32:49.487246 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:49.487344 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9 [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:49.488214 14100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:32:49.490892 14100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:32:49.496066 14100 catalog_manager.cc:1347] Generated new cluster ID: 794234a470c1463dbb27b60ca0e89bc5
I20260620 04:32:49.496145 14100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:32:49.513278 14100 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:32:49.514701 14100 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:32:49.524008 14100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd91c2c0fa8b4803b0836d38f119ace9: Generated new TSK 0
I20260620 04:32:49.524953 14100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:32:49.539064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 14016
2026-06-20T04:32:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1344 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:32:49 dist-test-slave-rzqm krb5kdc[14111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:32:49 dist-test-slave-rzqm krb5kdc[14111](info): set up 2 sockets
Jun 20 04:32:49 dist-test-slave-rzqm krb5kdc[14111](info): commencing operation
krb5kdc: starting...
W20260620 04:32:50.501039 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:32:50 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:32:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:32:50.624953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:50.733537 14124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:50.733803 14124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:50.733842 14124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:50.737784 14124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:32:50.737846 14124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:50.737864 14124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:50.737879 14124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:32:50.737892 14124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:32:50.743181 14124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:50.744225 14124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:50.745232 14124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:50.753521 14129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:50.753705 14124 server_base.cc:1034] running on GCE node
W20260620 04:32:50.753551 14130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:50.753547 14132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:50.754426 14124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:50.755553 14124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:50.756749 14124 hybrid_clock.cc:648] HybridClock initialized: now 1781929970756710 us; error 54 us; skew 500 ppm
Jun 20 04:32:50 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:50.759752 14124 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:32:50.760751 14124 webserver.cc:460] Webserver started at http://127.0.98.62:35633/ using document root <none> and password file <none>
I20260620 04:32:50.761369 14124 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:50.761442 14124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:50.761662 14124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:32:50.763779 14124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd"
format_stamp: "Formatted at 2026-06-20 04:32:50 on dist-test-slave-rzqm"
server_key: "4306e1b93d47cfd5f974198ff2aa21a8"
server_key_iv: "e78a57fe1bf7fd1903a8ebaf260f59d4"
server_key_version: "encryptionkey@0"
I20260620 04:32:50.764350 14124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd"
format_stamp: "Formatted at 2026-06-20 04:32:50 on dist-test-slave-rzqm"
server_key: "4306e1b93d47cfd5f974198ff2aa21a8"
server_key_iv: "e78a57fe1bf7fd1903a8ebaf260f59d4"
server_key_version: "encryptionkey@0"
I20260620 04:32:50.768179 14124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:32:50.770907 14139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:50.772195 14124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:32:50.772348 14124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd"
format_stamp: "Formatted at 2026-06-20 04:32:50 on dist-test-slave-rzqm"
server_key: "4306e1b93d47cfd5f974198ff2aa21a8"
server_key_iv: "e78a57fe1bf7fd1903a8ebaf260f59d4"
server_key_version: "encryptionkey@0"
I20260620 04:32:50.772451 14124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:50.786120 14124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:50.789419 14124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:50.789595 14124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:50.796311 14124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37239
I20260620 04:32:50.796622 14191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37239 every 8 connection(s)
I20260620 04:32:50.797312 14124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:32:50.800062 14192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:50.801190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 14124
I20260620 04:32:50.801294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:32:50.801594 392 external_mini_cluster.cc:1428] Setting key 692ccb93176de5ffd35e33a5d8800b82
I20260620 04:32:50.806633 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Bootstrap starting.
I20260620 04:32:50.809219 14192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:50.810027 14192 log.cc:824] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:32:50 dist-test-slave-rzqm krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:50.812211 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: No bootstrap required, opened a new log
I20260620 04:32:50.816015 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:50.802912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.98.62:37239)
0620 04:32:50.803320 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:50.803338 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:50.803382 (+ 44us) server_negotiation.cc:407] Connection header received
0620 04:32:50.804210 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:50.804233 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:50.804643 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:50.805052 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:50.805717 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:50.807132 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:50.807980 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:50.808665 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:50.808830 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:50.811507 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:50.811515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:50.811536 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:50.811571 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:50.813242 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:50.813723 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:50.813728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:50.813732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:50.813885 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:50.814172 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:50.814173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:50.814175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:50.814557 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:50.814703 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:50.814991 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 04:32:50.815245 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":103,"threads_started":1}
I20260620 04:32:50.816128 14192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:50.816382 14192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:50.816437 14192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1e7832e2e6456381c07c6f8e3f8fbd, State: Initialized, Role: FOLLOWER
I20260620 04:32:50.816869 14192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:50.817018 14192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:32:50.817111 14192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:32:50.817199 14192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:50.818073 14192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:50.818399 14192 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f1e7832e2e6456381c07c6f8e3f8fbd; no voters:
I20260620 04:32:50.818686 14192 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:32:50.818794 14197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:32:50.819018 14197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 1 LEADER]: Becoming Leader. State: Replica: 7f1e7832e2e6456381c07c6f8e3f8fbd, State: Running, Role: LEADER
I20260620 04:32:50.819326 14197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:50.819825 14192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:32:50.820887 14198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } } }
I20260620 04:32:50.821038 14198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:50.821413 14206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:32:50.821379 14199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1e7832e2e6456381c07c6f8e3f8fbd. Latest consensus state: current_term: 1 leader_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } } }
I20260620 04:32:50.822082 14199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:50.824347 14206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:32:50.830181 14206 catalog_manager.cc:1347] Generated new cluster ID: 7d74f7cca0c54952ac60e92c98183965
I20260620 04:32:50.830462 14206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:32:50.844924 14206 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:32:50.846378 14206 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:32:50.853677 14206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Generated new TSK 0
I20260620 04:32:50.854516 14206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:32:50.888109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:51.002942 14218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:51.003221 14218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:51.003281 14218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:51.007452 14218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:51.007529 14218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:51.007615 14218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:32:51.013095 14218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:51.014227 14218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:51.015235 14218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:51.024973 14226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:51.024948 14223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:51.025054 14218 server_base.cc:1034] running on GCE node
W20260620 04:32:51.024977 14224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:51.025643 14218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:51.026263 14218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:51.027436 14218 hybrid_clock.cc:648] HybridClock initialized: now 1781929971027405 us; error 44 us; skew 500 ppm
Jun 20 04:32:51 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:51.030225 14218 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:32:51.031178 14218 webserver.cc:460] Webserver started at http://127.0.98.1:41843/ using document root <none> and password file <none>
I20260620 04:32:51.031718 14218 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:51.031780 14218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:51.031957 14218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:32:51.034049 14218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "d37fdc9ed89d471e9426c13b2af2f87b"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "9b9ffc081708367ae8d41f01adfeec82"
server_key_iv: "a4c53f0626a6e4146933654de0a9df28"
server_key_version: "encryptionkey@0"
I20260620 04:32:51.034564 14218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d37fdc9ed89d471e9426c13b2af2f87b"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "9b9ffc081708367ae8d41f01adfeec82"
server_key_iv: "a4c53f0626a6e4146933654de0a9df28"
server_key_version: "encryptionkey@0"
I20260620 04:32:51.038328 14218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:32:51.040871 14233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:51.042359 14218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:32:51.042481 14218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "d37fdc9ed89d471e9426c13b2af2f87b"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "9b9ffc081708367ae8d41f01adfeec82"
server_key_iv: "a4c53f0626a6e4146933654de0a9df28"
server_key_version: "encryptionkey@0"
I20260620 04:32:51.042587 14218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:51.055897 14218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:51.058844 14218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:51.059005 14218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:51.059530 14218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:51.060340 14218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:51.060388 14218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:51.060427 14218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:51.060448 14218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:51.069778 14218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40883
I20260620 04:32:51.069864 14346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40883 every 8 connection(s)
I20260620 04:32:51.070720 14218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:32:51.075022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 14218
I20260620 04:32:51.075169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:32:51.075465 392 external_mini_cluster.cc:1428] Setting key b1b5d6223d221c50c2fe352b87d4c6a8
Jun 20 04:32:51 dist-test-slave-rzqm krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:32:51.085373 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.072644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36069 (local address 127.0.98.62:37239)
0620 04:32:51.072818 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:51.072822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:51.073977 (+ 1155us) server_negotiation.cc:407] Connection header received
0620 04:32:51.074873 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:51.074876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:51.074931 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:51.075004 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:51.076338 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.076964 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.078094 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.078337 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.078422 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.081161 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:51.081165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:51.081168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:51.081199 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:51.082955 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:51.083497 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:51.083501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:51.083503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:51.083595 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:51.084033 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:51.084036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:51.084038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:51.084286 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:51.084397 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:51.085106 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 04:32:51.085233 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":72}
I20260620 04:32:51.086262 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.073065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:36069)
0620 04:32:51.073767 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:51.073855 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:51.074663 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:51.075193 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:51.075202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:51.075660 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:51.076187 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.076202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.077086 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.077089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.077976 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.077985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.078480 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.078484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.078648 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.079278 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:51.079294 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:51.080905 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:51.083094 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:51.083100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:51.083114 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:51.083390 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:51.083718 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:51.083721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:51.083722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:51.083924 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:51.084434 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:51.084440 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:51.084714 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 04:32:51.084993 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 04:32:51.085337 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:32:51.087580 14347 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37239
I20260620 04:32:51.087824 14347 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:51.088327 14347 heartbeater.cc:511] Master 127.0.98.62:37239 requested a full tablet report, sending...
I20260620 04:32:51.089905 14156 ts_manager.cc:194] Registered new tserver with Master: d37fdc9ed89d471e9426c13b2af2f87b (127.0.98.1:40883)
I20260620 04:32:51.091073 14156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:32:51.104509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:51.215742 14352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:51.216027 14352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:51.216070 14352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:51.220211 14352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:51.220283 14352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:51.220345 14352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:32:51.225876 14352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:51.226961 14352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:51.227967 14352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:51.237756 14358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:51.237777 14357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:51.237830 14352 server_base.cc:1034] running on GCE node
W20260620 04:32:51.237759 14360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:51.238431 14352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:51.239096 14352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:51.240278 14352 hybrid_clock.cc:648] HybridClock initialized: now 1781929971240258 us; error 32 us; skew 500 ppm
Jun 20 04:32:51 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:51.243132 14352 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:32:51.244135 14352 webserver.cc:460] Webserver started at http://127.0.98.2:44585/ using document root <none> and password file <none>
I20260620 04:32:51.244704 14352 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:51.244782 14352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:51.244976 14352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:32:51.247242 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "ceb2561b7233497aa9a3680a7dc196c8"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "6f7d60c48624067841cbfe62b1c44aa8"
server_key_iv: "3ded53bf6e455a24ea98e38f1c95badd"
server_key_version: "encryptionkey@0"
I20260620 04:32:51.247818 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ceb2561b7233497aa9a3680a7dc196c8"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "6f7d60c48624067841cbfe62b1c44aa8"
server_key_iv: "3ded53bf6e455a24ea98e38f1c95badd"
server_key_version: "encryptionkey@0"
I20260620 04:32:51.251883 14352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 04:32:51.254608 14367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:51.255978 14352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:32:51.256115 14352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "ceb2561b7233497aa9a3680a7dc196c8"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "6f7d60c48624067841cbfe62b1c44aa8"
server_key_iv: "3ded53bf6e455a24ea98e38f1c95badd"
server_key_version: "encryptionkey@0"
I20260620 04:32:51.256238 14352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:51.292078 14352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:51.295410 14352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:51.295591 14352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:51.296245 14352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:51.297191 14352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:51.297245 14352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:51.297286 14352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:51.297299 14352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:51.306396 14352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36363
I20260620 04:32:51.306473 14480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36363 every 8 connection(s)
I20260620 04:32:51.307341 14352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:32:51.311941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 14352
I20260620 04:32:51.312072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:32:51.312420 392 external_mini_cluster.cc:1428] Setting key 45574aeeac0e2c526be1d4489bee6082
Jun 20 04:32:51 dist-test-slave-rzqm krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:32:51.323048 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.309232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57843 (local address 127.0.98.62:37239)
0620 04:32:51.309397 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:51.309402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:51.310377 (+ 975us) server_negotiation.cc:407] Connection header received
0620 04:32:51.311883 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:51.311887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:51.311966 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:51.312061 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:51.313733 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.314335 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.316027 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.316273 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.316368 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.319169 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:51.319172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:51.319174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:51.319201 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:51.320613 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:51.321281 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:51.321284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:51.321286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:51.321366 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:51.321761 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:51.321764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:51.321767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:51.322043 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:51.322159 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:51.322744 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 04:32:51.322879 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:32:51.323969 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.309668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:57843)
0620 04:32:51.310192 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:51.310237 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:51.311583 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:51.312403 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:51.312413 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:51.312983 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:51.313596 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.313610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.314493 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.314496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.315898 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.315910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.316434 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.316438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.316623 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.317245 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:51.317262 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:51.318979 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:51.320733 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:51.320739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:51.320765 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:51.321177 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:51.321454 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:51.321457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:51.321459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:51.321603 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:51.322176 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:51.322183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:51.322491 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 04:32:51.322665 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 04:32:51.323018 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:32:51.325424 14481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37239
I20260620 04:32:51.325676 14481 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:51.326188 14481 heartbeater.cc:511] Master 127.0.98.62:37239 requested a full tablet report, sending...
I20260620 04:32:51.327263 14156 ts_manager.cc:194] Registered new tserver with Master: ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363)
I20260620 04:32:51.327849 14156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:32:51.340104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:51.450557 14486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:51.450845 14486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:51.450887 14486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:51.455101 14486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:51.455176 14486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:51.455237 14486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:32:51.460654 14486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:51.461752 14486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:51.462713 14486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:51.472196 14494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:51.472201 14491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:51.472201 14492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:51.472307 14486 server_base.cc:1034] running on GCE node
I20260620 04:32:51.472922 14486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:51.473572 14486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:51.474736 14486 hybrid_clock.cc:648] HybridClock initialized: now 1781929971474713 us; error 33 us; skew 500 ppm
Jun 20 04:32:51 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:51.477666 14486 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:32:51.478592 14486 webserver.cc:460] Webserver started at http://127.0.98.3:35129/ using document root <none> and password file <none>
I20260620 04:32:51.479171 14486 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:51.479249 14486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:51.479460 14486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:32:51.481662 14486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "28457bdb3b294aa49f6ebf41f837b8eb"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "9e152fd15ff1c83165c1bd0d531567bb"
server_key_iv: "24ab378683c81cd9dd8693d659943d84"
server_key_version: "encryptionkey@0"
I20260620 04:32:51.482235 14486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "28457bdb3b294aa49f6ebf41f837b8eb"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "9e152fd15ff1c83165c1bd0d531567bb"
server_key_iv: "24ab378683c81cd9dd8693d659943d84"
server_key_version: "encryptionkey@0"
I20260620 04:32:51.486150 14486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:32:51.488677 14501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:51.490175 14486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:32:51.490312 14486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "28457bdb3b294aa49f6ebf41f837b8eb"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "9e152fd15ff1c83165c1bd0d531567bb"
server_key_iv: "24ab378683c81cd9dd8693d659943d84"
server_key_version: "encryptionkey@0"
I20260620 04:32:51.490419 14486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:51.504631 14486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:51.507894 14486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:51.508081 14486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:51.508674 14486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:51.509578 14486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:51.509627 14486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:51.509699 14486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:51.509737 14486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:51.520112 14486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46087
I20260620 04:32:51.521536 14486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:32:51.521562 14614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46087 every 8 connection(s)
I20260620 04:32:51.526592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 14486
I20260620 04:32:51.526726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:32:51.526997 392 external_mini_cluster.cc:1428] Setting key b43f05fb75dbe21b4feb9727793f4d91
Jun 20 04:32:51 dist-test-slave-rzqm krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:51.539413 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.523947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52241 (local address 127.0.98.62:37239)
0620 04:32:51.524108 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:51.524113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:51.525347 (+ 1234us) server_negotiation.cc:407] Connection header received
0620 04:32:51.526301 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:51.526305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:51.526364 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:51.526438 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:51.528435 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.529286 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.531303 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.531645 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.531722 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.535030 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:51.535035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:51.535039 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:51.535075 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:51.536745 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:51.537474 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:51.537478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:51.537480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:51.537578 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:51.538024 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:51.538028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:51.538029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:51.538261 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:51.538345 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:51.539090 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 04:32:51.539238 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:32:51.540338 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.524411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:52241)
0620 04:32:51.525156 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:51.525221 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:51.526073 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:51.526894 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:51.526904 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:51.527423 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:51.528256 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.528279 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.529604 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.529608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.531134 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.531149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.531969 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.531973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.532179 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.532848 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:51.532865 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:51.534809 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:51.536889 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:51.536896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:51.536915 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:51.537365 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:51.537705 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:51.537708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:51.537709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:51.537905 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:51.538408 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:51.538415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:51.538790 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 04:32:51.539016 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 04:32:51.539388 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":122,"threads_started":1}
I20260620 04:32:51.541926 14615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37239
I20260620 04:32:51.542465 14615 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:51.543085 14615 heartbeater.cc:511] Master 127.0.98.62:37239 requested a full tablet report, sending...
I20260620 04:32:51.544246 14156 ts_manager.cc:194] Registered new tserver with Master: 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087)
I20260620 04:32:51.544896 14156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52241
I20260620 04:32:51.555176 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:32:51.564762 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.557180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.0.98.62:37239)
0620 04:32:51.557296 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:51.557301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:51.557507 (+ 206us) server_negotiation.cc:407] Connection header received
0620 04:32:51.557605 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:51.557607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:51.557647 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:51.557713 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:51.558397 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.558943 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.559883 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.560120 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.560205 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.561701 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:51.561705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:51.561708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:51.561797 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:51.563110 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:51.563528 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:51.563531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:51.563533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:51.563612 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:51.563863 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:51.563866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:51.563868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:51.564094 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:51.564211 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:51.564472 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 04:32:51.564580 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 04:32:51.567847 14156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:32:51.570971 14156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:32:51.586210 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.580625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.0.98.1:40883)
0620 04:32:51.581130 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:51.581145 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:51.581168 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:32:51.581248 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:51.581253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:51.581510 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:51.581694 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:51.582009 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.583156 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.584860 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.585498 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.585600 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.585762 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:51.585948 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 04:32:51.586052 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":138,"threads_started":1}
I20260620 04:32:51.586215 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.580705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46087 (local address 127.0.0.1:37940)
0620 04:32:51.581425 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:51.581438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:51.581533 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:51.582108 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:51.582112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:51.582130 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:51.582246 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.582253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.583597 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.583601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.584575 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.584585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.585834 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.585838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.585937 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.585959 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:32:51.586016 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 04:32:51.586068 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260620 04:32:51.586242 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.579892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40883 (local address 127.0.0.1:56790)
0620 04:32:51.580250 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:51.580305 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:51.580449 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:51.581681 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:51.581687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:51.581707 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:51.581877 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.581885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.583286 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.583290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.584699 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.584711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.585619 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.585623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.585742 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.585783 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 04:32:51.585906 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 04:32:51.586040 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":79,"thread_start_us":90,"threads_started":1}
I20260620 04:32:51.586535 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.580021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.0.98.2:36363)
0620 04:32:51.581132 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:51.581145 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:51.581501 (+ 356us) server_negotiation.cc:407] Connection header received
0620 04:32:51.581586 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:51.581591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:51.581782 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:51.581936 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:51.582159 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.582962 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.584823 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.585618 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.585702 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.585812 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:51.586281 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 04:32:51.586405 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":140,"threads_started":1}
I20260620 04:32:51.586779 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.580887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.0.98.3:46087)
0620 04:32:51.581288 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:51.581298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:51.581541 (+ 243us) server_negotiation.cc:407] Connection header received
0620 04:32:51.581616 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:51.581620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:51.581782 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:51.581941 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:51.582374 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.583457 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.584700 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.585555 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.585660 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.585812 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:51.586566 (+ 754us) server_negotiation.cc:299] Negotiation successful
0620 04:32:51.586669 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":142,"threads_started":1}
I20260620 04:32:51.587868 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.579895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36363 (local address 127.0.0.1:33716)
0620 04:32:51.581286 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:51.581300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:51.581418 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:51.581921 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:51.581924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:51.581941 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:51.582034 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.582039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.583097 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.583101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.584710 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.584716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.585876 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.585880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.585972 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.585995 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:32:51.586061 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 04:32:51.586130 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"mutex_wait_us":97,"thread_start_us":84,"threads_started":1}
I20260620 04:32:51.590132 14415 tablet_service.cc:1467] Processing CreateTablet for tablet fcbea2b2b7364bca8c04de65c0a44c10 (DEFAULT_TABLE table=test-table [id=26fab7e926044703b28ed6409825033b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:32:51.590019 14549 tablet_service.cc:1467] Processing CreateTablet for tablet fcbea2b2b7364bca8c04de65c0a44c10 (DEFAULT_TABLE table=test-table [id=26fab7e926044703b28ed6409825033b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:32:51.590018 14281 tablet_service.cc:1467] Processing CreateTablet for tablet fcbea2b2b7364bca8c04de65c0a44c10 (DEFAULT_TABLE table=test-table [id=26fab7e926044703b28ed6409825033b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:32:51.591132 14281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbea2b2b7364bca8c04de65c0a44c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:51.591136 14549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbea2b2b7364bca8c04de65c0a44c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:51.591176 14415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbea2b2b7364bca8c04de65c0a44c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:51.597644 14632 tablet_bootstrap.cc:492] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: Bootstrap starting.
I20260620 04:32:51.599128 14633 tablet_bootstrap.cc:492] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Bootstrap starting.
I20260620 04:32:51.600126 14632 tablet_bootstrap.cc:654] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:51.600844 14632 log.cc:824] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:51.600878 14633 tablet_bootstrap.cc:654] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:51.601363 14634 tablet_bootstrap.cc:492] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Bootstrap starting.
I20260620 04:32:51.601713 14633 log.cc:824] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:51.602923 14632 tablet_bootstrap.cc:492] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: No bootstrap required, opened a new log
I20260620 04:32:51.603107 14632 ts_tablet_manager.cc:1397] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:32:51.603541 14634 tablet_bootstrap.cc:654] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:51.603797 14633 tablet_bootstrap.cc:492] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: No bootstrap required, opened a new log
I20260620 04:32:51.603993 14633 ts_tablet_manager.cc:1397] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:32:51.604733 14634 log.cc:824] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:51.606339 14632 raft_consensus.cc:348] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:51.606571 14632 raft_consensus.cc:374] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:51.606647 14632 raft_consensus.cc:729] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb2561b7233497aa9a3680a7dc196c8, State: Initialized, Role: FOLLOWER
I20260620 04:32:51.606830 14634 tablet_bootstrap.cc:492] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: No bootstrap required, opened a new log
I20260620 04:32:51.607101 14634 ts_tablet_manager.cc:1397] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:32:51.607079 14632 consensus_queue.cc:260] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:51.607681 14481 heartbeater.cc:503] Master 127.0.98.62:37239 was elected leader, sending a full tablet report...
I20260620 04:32:51.608029 14632 ts_tablet_manager.cc:1428] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:32:51.608245 14633 raft_consensus.cc:348] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:51.608541 14633 raft_consensus.cc:374] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:51.608605 14633 raft_consensus.cc:729] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37fdc9ed89d471e9426c13b2af2f87b, State: Initialized, Role: FOLLOWER
I20260620 04:32:51.609094 14633 consensus_queue.cc:260] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:51.609814 14347 heartbeater.cc:503] Master 127.0.98.62:37239 was elected leader, sending a full tablet report...
I20260620 04:32:51.610179 14633 ts_tablet_manager.cc:1428] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:32:51.610980 14634 raft_consensus.cc:348] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:51.611203 14634 raft_consensus.cc:374] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:51.611281 14634 raft_consensus.cc:729] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28457bdb3b294aa49f6ebf41f837b8eb, State: Initialized, Role: FOLLOWER
I20260620 04:32:51.611721 14634 consensus_queue.cc:260] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:51.612346 14615 heartbeater.cc:503] Master 127.0.98.62:37239 was elected leader, sending a full tablet report...
I20260620 04:32:51.612692 14634 ts_tablet_manager.cc:1428] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:32:51.715040 14639 raft_consensus.cc:482] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:32:51.715255 14639 raft_consensus.cc:504] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:51.716379 14639 leader_election.cc:288] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363), 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087)
I20260620 04:32:51.720494 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.716840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36295 (local address 127.0.98.3:46087)
0620 04:32:51.716972 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:51.716977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:51.717220 (+ 243us) server_negotiation.cc:407] Connection header received
0620 04:32:51.717361 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:51.717364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:51.717410 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:51.717469 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:51.717768 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.718252 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.719447 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.719928 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.720023 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.720089 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:51.720279 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 04:32:51.720348 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:32:51.720503 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.716841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53773 (local address 127.0.98.2:36363)
0620 04:32:51.716971 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:51.716975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:51.717183 (+ 208us) server_negotiation.cc:407] Connection header received
0620 04:32:51.717319 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:51.717323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:51.717366 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:51.717430 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:51.717713 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.718209 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.719449 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.719956 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.720048 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.720109 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:51.720297 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 04:32:51.720363 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:32:51.720516 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.716687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46087 (local address 127.0.98.1:36295)
0620 04:32:51.717123 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:51.717134 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:51.717260 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:51.717495 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:51.717498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:51.717513 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:51.717663 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.717668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.718360 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.718363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.719322 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.719332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.720043 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.720046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.720150 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.720173 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:32:51.720246 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 04:32:51.720335 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":85,"thread_start_us":110,"threads_started":1}
I20260620 04:32:51.720526 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.716652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36363 (local address 127.0.98.1:53773)
0620 04:32:51.717090 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:51.717102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:51.717210 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:51.717452 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:51.717455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:51.717469 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:51.717610 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.717615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.718322 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.718325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.719313 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:51.719323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.720069 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.720073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:51.720185 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.720204 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:32:51.720269 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 04:32:51.720352 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":29,"thread_start_us":132,"threads_started":1}
I20260620 04:32:51.721297 14435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbea2b2b7364bca8c04de65c0a44c10" candidate_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb2561b7233497aa9a3680a7dc196c8" is_pre_election: true
I20260620 04:32:51.721360 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbea2b2b7364bca8c04de65c0a44c10" candidate_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" is_pre_election: true
I20260620 04:32:51.721608 14435 raft_consensus.cc:2454] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37fdc9ed89d471e9426c13b2af2f87b in term 0.
I20260620 04:32:51.721637 14563 raft_consensus.cc:2454] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37fdc9ed89d471e9426c13b2af2f87b in term 0.
I20260620 04:32:51.722092 14236 leader_election.cc:302] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ceb2561b7233497aa9a3680a7dc196c8, d37fdc9ed89d471e9426c13b2af2f87b; no voters:
I20260620 04:32:51.722338 14639 raft_consensus.cc:2781] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:32:51.722424 14639 raft_consensus.cc:482] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:32:51.722476 14639 raft_consensus.cc:3037] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:51.723492 14639 raft_consensus.cc:504] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:51.723876 14639 leader_election.cc:288] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 1 election: Requested vote from peers ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363), 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087)
I20260620 04:32:51.724296 14435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbea2b2b7364bca8c04de65c0a44c10" candidate_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb2561b7233497aa9a3680a7dc196c8"
I20260620 04:32:51.724296 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbea2b2b7364bca8c04de65c0a44c10" candidate_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28457bdb3b294aa49f6ebf41f837b8eb"
I20260620 04:32:51.724414 14563 raft_consensus.cc:3037] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:51.724421 14435 raft_consensus.cc:3037] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:51.725551 14563 raft_consensus.cc:2454] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37fdc9ed89d471e9426c13b2af2f87b in term 1.
I20260620 04:32:51.725576 14435 raft_consensus.cc:2454] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37fdc9ed89d471e9426c13b2af2f87b in term 1.
I20260620 04:32:51.725840 14235 leader_election.cc:302] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28457bdb3b294aa49f6ebf41f837b8eb, d37fdc9ed89d471e9426c13b2af2f87b; no voters:
I20260620 04:32:51.726003 14639 raft_consensus.cc:2781] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:32:51.726222 14639 raft_consensus.cc:686] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 LEADER]: Becoming Leader. State: Replica: d37fdc9ed89d471e9426c13b2af2f87b, State: Running, Role: LEADER
I20260620 04:32:51.726599 14639 consensus_queue.cc:237] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:51.730229 14156 catalog_manager.cc:5495] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b reported cstate change: term changed from 0 to 1, leader changed from <none> to d37fdc9ed89d471e9426c13b2af2f87b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } health_report { overall_health: UNKNOWN } } }
W20260620 04:32:51.772830 14616 tablet.cc:2369] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:32:51.776503 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.772883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.0.98.1:40883)
0620 04:32:51.773028 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:51.773033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:51.773096 (+ 63us) server_negotiation.cc:407] Connection header received
0620 04:32:51.773269 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:51.773272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:51.773337 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:51.773430 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:32:51.773662 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.774168 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.775005 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:51.775224 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:51.775315 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:51.775656 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:32:51.776091 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:32:51.776221 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:51.776299 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 04:32:51.776362 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:32:51.782761 14435 raft_consensus.cc:1261] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8 [term 1 FOLLOWER]: Refusing update from remote peer d37fdc9ed89d471e9426c13b2af2f87b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:32:51.782806 14563 raft_consensus.cc:1261] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb [term 1 FOLLOWER]: Refusing update from remote peer d37fdc9ed89d471e9426c13b2af2f87b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:32:51.783538 14643 consensus_queue.cc:1035] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:32:51.783772 14639 consensus_queue.cc:1035] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:32:51.791043 14646 mvcc.cc:205] Tried to move back new op lower bound from 7298785164414230528 to 7298785164195008512. Current Snapshot: MvccSnapshot[applied={T|T < 7298785164414230528}]
I20260620 04:32:51.793653 14647 mvcc.cc:205] Tried to move back new op lower bound from 7298785164414230528 to 7298785164195008512. Current Snapshot: MvccSnapshot[applied={T|T < 7298785164414230528}]
I20260620 04:32:51.803102 14153 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:32:51.803289 14153 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:32:51.806205 14153 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Sending DeleteTablet for 3 replicas of tablet fcbea2b2b7364bca8c04de65c0a44c10
I20260620 04:32:51.807164 14415 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbea2b2b7364bca8c04de65c0a44c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:32:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33716
I20260620 04:32:51.807260 14549 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbea2b2b7364bca8c04de65c0a44c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:32:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37940
I20260620 04:32:51.807896 14281 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbea2b2b7364bca8c04de65c0a44c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:32:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56790
I20260620 04:32:51.808050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 14218
I20260620 04:32:51.809373 14655 tablet_replica.cc:330] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: stopping tablet replica
I20260620 04:32:51.809850 14655 raft_consensus.cc:2229] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:32:51.810195 14655 raft_consensus.cc:2258] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:32:51.811722 14655 ts_tablet_manager.cc:1905] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:32:51.814334 14656 tablet_replica.cc:330] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: stopping tablet replica
I20260620 04:32:51.814404 14655 ts_tablet_manager.cc:1918] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 04:32:51.814421 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40883 (error 108)
I20260620 04:32:51.814558 14655 log.cc:1196] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/fcbea2b2b7364bca8c04de65c0a44c10
I20260620 04:32:51.814620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 14352
I20260620 04:32:51.814908 14655 ts_tablet_manager.cc:1939] T fcbea2b2b7364bca8c04de65c0a44c10 P ceb2561b7233497aa9a3680a7dc196c8: Deleting consensus metadata
I20260620 04:32:51.816464 14656 raft_consensus.cc:2229] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:32:51.816843 14656 raft_consensus.cc:2258] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:32:51.818455 14656 ts_tablet_manager.cc:1905] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:32:51.818641 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.818326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40883 (local address 127.0.0.1:56814)
0620 04:32:51.818469 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:51.818553 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 04:32:51.818853 14141 catalog_manager.cc:4584] TS d37fdc9ed89d471e9426c13b2af2f87b (127.0.98.1:40883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fcbea2b2b7364bca8c04de65c0a44c10: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40883: connect: Connection refused (error 111)
I20260620 04:32:51.820518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 14486
I20260620 04:32:51.821923 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:51.820785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36363 (local address 127.0.0.1:33724)
0620 04:32:51.821780 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:51.821840 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":898}
I20260620 04:32:51.826079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 14124
I20260620 04:32:51.832023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37239
--webserver_interface=127.0.98.62
--webserver_port=35633
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:51.944657 14658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:51.944958 14658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:51.945036 14658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:51.949245 14658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:32:51.949316 14658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:51.949335 14658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:51.949352 14658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:32:51.949365 14658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:32:51.955036 14658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:51.956194 14658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:51.957319 14658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:51.970460 14667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:51.970472 14665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:51.970484 14664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:51.970762 14658 server_base.cc:1034] running on GCE node
I20260620 04:32:51.971431 14658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:51.972679 14658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:51.973858 14658 hybrid_clock.cc:648] HybridClock initialized: now 1781929971973815 us; error 62 us; skew 500 ppm
Jun 20 04:32:51 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:51.977154 14658 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:32:51.978145 14658 webserver.cc:460] Webserver started at http://127.0.98.62:35633/ using document root <none> and password file <none>
I20260620 04:32:51.978739 14658 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:51.978822 14658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:51.982926 14658 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:32:51.985530 14674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:51.987032 14658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:32:51.987182 14658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd"
format_stamp: "Formatted at 2026-06-20 04:32:50 on dist-test-slave-rzqm"
server_key: "4306e1b93d47cfd5f974198ff2aa21a8"
server_key_iv: "e78a57fe1bf7fd1903a8ebaf260f59d4"
server_key_version: "encryptionkey@0"
I20260620 04:32:51.987829 14658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:52.007488 14658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:52.011318 14658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:52.011505 14658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:52.018759 14658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37239
I20260620 04:32:52.019030 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37239 every 8 connection(s)
I20260620 04:32:52.019906 14658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:32:52.024523 14727 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:32:52.026441 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Bootstrap starting.
I20260620 04:32:52.028968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 14658
I20260620 04:32:52.029481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40883
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37239
--webserver_port=41843
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:32:52.045327 14727 log.cc:824] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:52.054225 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:32:52.054596 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Bootstrap complete.
I20260620 04:32:52.058710 14727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:52.059134 14727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1e7832e2e6456381c07c6f8e3f8fbd, State: Initialized, Role: FOLLOWER
I20260620 04:32:52.059600 14727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:52.059716 14727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:32:52.059796 14727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:32:52.059891 14727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:32:52.061887 14727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:52.062211 14727 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f1e7832e2e6456381c07c6f8e3f8fbd; no voters:
I20260620 04:32:52.062517 14727 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:32:52.062618 14731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:32:52.062816 14731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 2 LEADER]: Becoming Leader. State: Replica: 7f1e7832e2e6456381c07c6f8e3f8fbd, State: Running, Role: LEADER
I20260620 04:32:52.063189 14731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:52.063855 14727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:32:52.064729 14732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } } }
I20260620 04:32:52.064771 14733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1e7832e2e6456381c07c6f8e3f8fbd. Latest consensus state: current_term: 2 leader_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } } }
I20260620 04:32:52.064862 14732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:52.064878 14733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:52.068910 14746 catalog_manager.cc:1259] Loaded cluster ID: 7d74f7cca0c54952ac60e92c98183965
I20260620 04:32:52.068979 14746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: loading cluster ID for follower catalog manager: success
W20260620 04:32:52.069846 14746 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260620 04:32:52.069908 14746 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260620 04:32:52.069974 14740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:32:52.072376 14740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:32:52.072870 14740 catalog_manager.cc:1259] Loaded cluster ID: 7d74f7cca0c54952ac60e92c98183965
I20260620 04:32:52.072934 14740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:32:52.074020 14740 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 04:32:52.078445 14740 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 04:32:52.084755 14740 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:32:52.085906 14740 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Loaded TSK: 0
I20260620 04:32:52.087231 14740 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:32:52.150965 14729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:52.151252 14729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:52.151317 14729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:52.155507 14729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:52.155575 14729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:52.155658 14729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:32:52.161078 14729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41843
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:52.162099 14729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:52.163107 14729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:52.173106 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:52.173069 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:52.173216 14729 server_base.cc:1034] running on GCE node
W20260620 04:32:52.173375 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:52.173820 14729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:52.174489 14729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:52.175665 14729 hybrid_clock.cc:648] HybridClock initialized: now 1781929972175651 us; error 27 us; skew 500 ppm
Jun 20 04:32:52 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:52.183388 14729 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:32:52.184463 14729 webserver.cc:460] Webserver started at http://127.0.98.1:41843/ using document root <none> and password file <none>
I20260620 04:32:52.185060 14729 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:52.185163 14729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:52.189363 14729 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:32:52.191984 14764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:52.193540 14729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:32:52.193661 14729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "d37fdc9ed89d471e9426c13b2af2f87b"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "9b9ffc081708367ae8d41f01adfeec82"
server_key_iv: "a4c53f0626a6e4146933654de0a9df28"
server_key_version: "encryptionkey@0"
I20260620 04:32:52.194365 14729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:52.221113 14729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:52.224203 14729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:52.224390 14729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:52.224972 14729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:52.226104 14771 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:32:52.230412 14729 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:32:52.230489 14729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 04:32:52.230559 14729 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:32:52.232268 14729 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:32:52.232319 14729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 04:32:52.232678 14771 tablet_bootstrap.cc:492] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Bootstrap starting.
I20260620 04:32:52.241644 14729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40883
I20260620 04:32:52.241703 14878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40883 every 8 connection(s)
I20260620 04:32:52.242586 14729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:32:52.246873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 14729
I20260620 04:32:52.247385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36363
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37239
--webserver_port=44585
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:32:52.251564 14771 log.cc:824] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:32:52 dist-test-slave-rzqm krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:52.259704 14771 tablet_bootstrap.cc:492] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:32:52.260087 14771 tablet_bootstrap.cc:492] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Bootstrap complete.
I20260620 04:32:52.261343 14771 ts_tablet_manager.cc:1397] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.011s
I20260620 04:32:52.261869 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.244581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:45705)
0620 04:32:52.245256 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:52.245338 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:52.246448 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:52.248025 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:52.248038 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:52.248564 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:52.249311 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.249337 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.250758 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.250761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.252043 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.252055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.252852 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.252856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.253097 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.253789 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:52.253811 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:52.255709 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:52.258404 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:52.258410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:52.258423 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:52.258766 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:52.259162 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:52.259165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:52.259166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:52.259340 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:52.260005 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:52.260011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:52.260308 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 04:32:52.260486 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 04:32:52.260818 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":119,"threads_started":1}
I20260620 04:32:52.261871 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.244591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45705 (local address 127.0.98.62:37239)
0620 04:32:52.244978 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.245035 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.245475 (+ 440us) server_negotiation.cc:407] Connection header received
0620 04:32:52.247001 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.247019 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.247455 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.247896 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:52.249525 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.250596 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.252164 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.252700 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.252883 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.255960 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:52.255968 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:52.255987 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:52.256026 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:52.258257 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:52.258899 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:52.258905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:52.258911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:52.259062 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:52.259433 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:52.259436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:52.259438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:52.259892 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:52.260016 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.260659 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.260969 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":122,"threads_started":1}
I20260620 04:32:52.264153 14771 raft_consensus.cc:348] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:52.264637 14771 raft_consensus.cc:729] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37fdc9ed89d471e9426c13b2af2f87b, State: Initialized, Role: FOLLOWER
I20260620 04:32:52.264940 14879 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37239
I20260620 04:32:52.265214 14879 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:52.265179 14771 consensus_queue.cc:260] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } }
I20260620 04:32:52.265769 14879 heartbeater.cc:511] Master 127.0.98.62:37239 requested a full tablet report, sending...
I20260620 04:32:52.266219 14771 ts_tablet_manager.cc:1428] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:32:52.268420 14691 ts_manager.cc:194] Registered new tserver with Master: d37fdc9ed89d471e9426c13b2af2f87b (127.0.98.1:40883)
I20260620 04:32:52.272521 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45705
I20260620 04:32:52.275079 14879 heartbeater.cc:503] Master 127.0.98.62:37239 was elected leader, sending a full tablet report...
Jun 20 04:32:52 dist-test-slave-rzqm krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:32:52.283002 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.271792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40883 (local address 127.0.0.1:56826)
0620 04:32:52.272526 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:52.272570 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:52.272745 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:52.273286 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:52.273292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:52.273785 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:52.273919 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.273928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.275271 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.275275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.276200 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.276212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.277057 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.277076 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.277172 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.277746 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:52.277760 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:52.279103 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:52.281171 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:52.281180 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:52.281185 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:52.281438 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:52.281743 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:52.281745 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:52.281747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:52.281851 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:52.282404 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:52.282410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:52.282530 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 04:32:52.282665 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 04:32:52.282865 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":120,"threads_started":1}
I20260620 04:32:52.283087 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.271886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.0.98.1:40883)
0620 04:32:52.272288 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.272299 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.272869 (+ 570us) server_negotiation.cc:407] Connection header received
0620 04:32:52.272933 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.272937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.273143 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.273266 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:52.274045 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.275131 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.276334 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.276905 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.277038 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.279247 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:52.279257 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:52.279270 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:52.279298 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:52.280976 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:52.281548 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:52.281553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:52.281555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:52.281658 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:52.281928 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:52.281931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:52.281933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:52.282239 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:52.282388 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.282750 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.282921 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":88,"threads_started":1}
I20260620 04:32:52.284328 14813 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbea2b2b7364bca8c04de65c0a44c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:32:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56826
I20260620 04:32:52.285214 14888 tablet_replica.cc:330] stopping tablet replica
I20260620 04:32:52.285538 14888 raft_consensus.cc:2229] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:32:52.285815 14888 raft_consensus.cc:2258] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:32:52.286991 14888 ts_tablet_manager.cc:1905] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:32:52.291741 14888 ts_tablet_manager.cc:1918] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:32:52.291855 14888 log.cc:1196] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/fcbea2b2b7364bca8c04de65c0a44c10
I20260620 04:32:52.292150 14888 ts_tablet_manager.cc:1939] T fcbea2b2b7364bca8c04de65c0a44c10 P d37fdc9ed89d471e9426c13b2af2f87b: Deleting consensus metadata
I20260620 04:32:52.294229 14676 catalog_manager.cc:4841] TS d37fdc9ed89d471e9426c13b2af2f87b (127.0.98.1:40883): tablet fcbea2b2b7364bca8c04de65c0a44c10 (table test-table [id=26fab7e926044703b28ed6409825033b]) successfully deleted
W20260620 04:32:52.370725 14883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:52.370995 14883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:52.371042 14883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:52.375176 14883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:52.375241 14883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:52.375306 14883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:32:52.380733 14883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44585
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:52.381846 14883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:52.382893 14883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:52.392375 14894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:52.392520 14893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:52.392460 14896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:52.392696 14883 server_base.cc:1034] running on GCE node
I20260620 04:32:52.393137 14883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:52.393761 14883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:52.394943 14883 hybrid_clock.cc:648] HybridClock initialized: now 1781929972394927 us; error 31 us; skew 500 ppm
Jun 20 04:32:52 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:52.397722 14883 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:32:52.398603 14883 webserver.cc:460] Webserver started at http://127.0.98.2:44585/ using document root <none> and password file <none>
I20260620 04:32:52.399173 14883 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:52.399252 14883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:52.403210 14883 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:32:52.405521 14903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:52.406983 14883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:32:52.407109 14883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "ceb2561b7233497aa9a3680a7dc196c8"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "6f7d60c48624067841cbfe62b1c44aa8"
server_key_iv: "3ded53bf6e455a24ea98e38f1c95badd"
server_key_version: "encryptionkey@0"
I20260620 04:32:52.407680 14883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:52.423790 14883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:52.427430 14883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:52.427616 14883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:52.428256 14883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:52.429124 14883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:52.429179 14883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:52.429234 14883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:52.429261 14883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:52.437862 14883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36363
I20260620 04:32:52.437940 15016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36363 every 8 connection(s)
I20260620 04:32:52.438885 14883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:32:52.445297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 14883
I20260620 04:32:52.445724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46087
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37239
--webserver_port=35129
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:32:52 dist-test-slave-rzqm krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:52.452946 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.440716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33327 (local address 127.0.98.62:37239)
0620 04:32:52.440858 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.440862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.441869 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 04:32:52.443128 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.443131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.443210 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.443290 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:52.444664 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.445249 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.446551 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.446794 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.446889 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.449512 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:52.449515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:52.449517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:52.449547 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:52.450882 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:52.451405 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:52.451409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:52.451410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:52.451491 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:52.451851 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:52.451854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:52.451856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:52.452043 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:52.452129 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.452635 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.452777 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:32:52.453850 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.441187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:33327)
0620 04:32:52.441679 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:52.441724 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:52.442893 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:52.443503 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:52.443510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:52.443957 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:52.444493 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.444509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.445379 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.445382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.446415 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.446426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.446943 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.446947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.447119 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.447683 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:52.447707 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:52.449316 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:52.450996 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:52.451003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:52.451030 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:52.451307 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:52.451570 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:52.451573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:52.451575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:52.451760 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:52.452140 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:52.452148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:52.452406 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 04:32:52.452580 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 04:32:52.452940 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":115,"threads_started":1}
I20260620 04:32:52.454998 15017 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37239
I20260620 04:32:52.455262 15017 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:52.455821 15017 heartbeater.cc:511] Master 127.0.98.62:37239 requested a full tablet report, sending...
I20260620 04:32:52.457098 14691 ts_manager.cc:194] Registered new tserver with Master: ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363)
I20260620 04:32:52.457860 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33327
W20260620 04:32:52.558441 15020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:52.558718 15020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:52.558763 15020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:52.562814 15020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:52.562875 15020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:52.562933 15020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:32:52.568398 15020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35129
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:52.569531 15020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:52.570608 15020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:52.580438 15025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:52.580442 15026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:52.580438 15028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:52.580829 15020 server_base.cc:1034] running on GCE node
I20260620 04:32:52.581349 15020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:52.582067 15020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:52.583248 15020 hybrid_clock.cc:648] HybridClock initialized: now 1781929972583221 us; error 39 us; skew 500 ppm
Jun 20 04:32:52 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:52.586007 15020 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:32:52.586915 15020 webserver.cc:460] Webserver started at http://127.0.98.3:35129/ using document root <none> and password file <none>
I20260620 04:32:52.587471 15020 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:52.587554 15020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:52.591734 15020 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:32:52.594087 15035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:52.595345 15020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:32:52.595456 15020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "28457bdb3b294aa49f6ebf41f837b8eb"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "9e152fd15ff1c83165c1bd0d531567bb"
server_key_iv: "24ab378683c81cd9dd8693d659943d84"
server_key_version: "encryptionkey@0"
I20260620 04:32:52.595994 15020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:52.619623 15020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:52.623059 15020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:52.623258 15020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:52.623872 15020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:52.625063 15042 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 04:32:52.629138 15042 ts_tablet_manager.cc:1852] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 04:32:52.629567 15042 ts_tablet_manager.cc:1905] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:32:52.633436 15042 ts_tablet_manager.cc:1918] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:32:52.633540 15042 log.cc:1196] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/fcbea2b2b7364bca8c04de65c0a44c10
I20260620 04:32:52.633838 15042 ts_tablet_manager.cc:1939] T fcbea2b2b7364bca8c04de65c0a44c10 P 28457bdb3b294aa49f6ebf41f837b8eb: Deleting consensus metadata
I20260620 04:32:52.634121 15020 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 04:32:52.634197 15020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260620 04:32:52.634268 15020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:52.634294 15020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:52.642889 15020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46087
I20260620 04:32:52.642953 15149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46087 every 8 connection(s)
I20260620 04:32:52.643805 15020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:32:52.652724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 15020
Jun 20 04:32:52 dist-test-slave-rzqm krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:32:52.658978 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.645766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40565 (local address 127.0.98.62:37239)
0620 04:32:52.645929 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.645940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.646848 (+ 908us) server_negotiation.cc:407] Connection header received
0620 04:32:52.647815 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.647818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.647868 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.647951 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:52.649291 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.649887 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.651110 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.651337 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.651433 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.654650 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:52.654654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:52.654656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:52.654682 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:52.656238 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:52.656823 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:52.656826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:52.656828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:52.656932 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:52.657395 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:52.657399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:52.657401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:52.657636 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:52.657737 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.658585 (+ 848us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.658726 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 04:32:52.659901 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.646162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:40565)
0620 04:32:52.646683 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:52.646730 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:52.647588 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:52.648150 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:52.648158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:52.648512 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:52.649120 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.649137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.650022 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.650024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.650958 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.650969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.651510 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.651514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.651764 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.652452 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:52.652475 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:52.654430 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:52.656389 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:52.656395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:52.656411 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:52.656715 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:52.657040 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:52.657043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:52.657045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:52.657289 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:52.657771 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:52.657780 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:52.658234 (+ 454us) client_negotiation.cc:769] Sending connection context
0620 04:32:52.658479 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 04:32:52.658883 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260620 04:32:52.661161 15150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37239
I20260620 04:32:52.661453 15150 heartbeater.cc:464] Registering TS with master...
I20260620 04:32:52.662034 15150 heartbeater.cc:511] Master 127.0.98.62:37239 requested a full tablet report, sending...
I20260620 04:32:52.662238 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.653475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.0.98.62:37239)
0620 04:32:52.653744 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.653748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.653906 (+ 158us) server_negotiation.cc:407] Connection header received
0620 04:32:52.654069 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.654072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.654128 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.654211 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:52.654997 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.655681 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.656599 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.656865 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.656967 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.658692 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:52.658696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:52.658699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:52.658719 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:52.660299 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:52.660730 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:52.660733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:52.660734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:52.660814 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:52.661073 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:52.661075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:52.661077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:52.661294 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:52.661491 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.661970 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.662074 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
I20260620 04:32:52.663133 14691 ts_manager.cc:194] Registered new tserver with Master: 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087)
I20260620 04:32:52.663727 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40565
I20260620 04:32:52.663968 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:32:52.668462 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.664662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.0.98.62:37239)
0620 04:32:52.664797 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.664801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.665053 (+ 252us) server_negotiation.cc:407] Connection header received
0620 04:32:52.665223 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.665226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.665321 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.665418 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:32:52.665643 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.666195 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.667049 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.667294 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.667380 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.667688 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:32:52.668081 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:32:52.668186 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.668264 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.668329 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:32:52.669260 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:32:52.672511 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:32:52.681093 14813 tablet_service.cc:1467] Processing CreateTablet for tablet afc6496e9c154c27882b226ee2d53316 (DEFAULT_TABLE table=test-table [id=9bab6d6ddacd4238a74ca20d5d42c0ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:32:52.681864 14813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc6496e9c154c27882b226ee2d53316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:52.684516 14771 tablet_bootstrap.cc:492] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Bootstrap starting.
I20260620 04:32:52.684628 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.679443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36363 (local address 127.0.0.1:33738)
0620 04:32:52.679605 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:52.679620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:52.679710 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:52.680537 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:52.680540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:52.680555 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:52.680668 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.680675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.682026 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.682029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.683289 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.683300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.684168 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.684171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.684293 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.684340 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 04:32:52.684413 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 04:32:52.684499 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 04:32:52.684697 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.679670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.0.98.2:36363)
0620 04:32:52.680022 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.680033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.680058 (+ 25us) server_negotiation.cc:407] Connection header received
0620 04:32:52.680144 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.680150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.680399 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.680571 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:52.680797 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.681883 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.683448 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.684042 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.684140 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.684292 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.684458 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.684551 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
I20260620 04:32:52.685601 14771 tablet_bootstrap.cc:654] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:52.686933 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.680091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46087 (local address 127.0.0.1:37958)
0620 04:32:52.680324 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:52.680337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:52.680425 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:52.681460 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:52.681463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:52.681623 (+ 160us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:52.681718 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.681725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.683525 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.683528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.684872 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.685059 (+ 187us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.686503 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.686507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.686605 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.686632 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 04:32:52.686724 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 04:32:52.686798 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260620 04:32:52.687386 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.680333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.0.98.3:46087)
0620 04:32:52.680822 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.680836 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.680857 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:32:52.680930 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.680935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.681192 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.681339 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:52.682281 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.683378 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.685179 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.686365 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.686486 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.686650 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.687107 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.687221 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":156,"threads_started":1}
I20260620 04:32:52.687492 14771 tablet_bootstrap.cc:492] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: No bootstrap required, opened a new log
I20260620 04:32:52.687582 14771 ts_tablet_manager.cc:1397] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 04:32:52.687963 14771 raft_consensus.cc:348] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:52.688055 14771 raft_consensus.cc:374] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:52.688088 14771 raft_consensus.cc:729] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37fdc9ed89d471e9426c13b2af2f87b, State: Initialized, Role: FOLLOWER
I20260620 04:32:52.687909 14951 tablet_service.cc:1467] Processing CreateTablet for tablet afc6496e9c154c27882b226ee2d53316 (DEFAULT_TABLE table=test-table [id=9bab6d6ddacd4238a74ca20d5d42c0ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:32:52.688392 14771 consensus_queue.cc:260] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:52.688683 14771 ts_tablet_manager.cc:1428] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 04:32:52.689095 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc6496e9c154c27882b226ee2d53316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:52.689754 15084 tablet_service.cc:1467] Processing CreateTablet for tablet afc6496e9c154c27882b226ee2d53316 (DEFAULT_TABLE table=test-table [id=9bab6d6ddacd4238a74ca20d5d42c0ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:32:52.690644 15084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc6496e9c154c27882b226ee2d53316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:32:52.694849 15042 tablet_bootstrap.cc:492] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: Bootstrap starting.
I20260620 04:32:52.696622 15163 tablet_bootstrap.cc:492] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: Bootstrap starting.
I20260620 04:32:52.697153 15042 tablet_bootstrap.cc:654] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:52.697923 15042 log.cc:824] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:52.699158 15163 tablet_bootstrap.cc:654] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: Neither blocks nor log segments found. Creating new log.
I20260620 04:32:52.699685 15042 tablet_bootstrap.cc:492] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: No bootstrap required, opened a new log
I20260620 04:32:52.699872 15042 ts_tablet_manager.cc:1397] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 04:32:52.699934 15163 log.cc:824] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:52.701778 15163 tablet_bootstrap.cc:492] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: No bootstrap required, opened a new log
I20260620 04:32:52.701949 15163 ts_tablet_manager.cc:1397] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:32:52.703011 15042 raft_consensus.cc:348] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:52.703342 15042 raft_consensus.cc:374] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:52.703449 15042 raft_consensus.cc:729] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28457bdb3b294aa49f6ebf41f837b8eb, State: Initialized, Role: FOLLOWER
I20260620 04:32:52.703872 15042 consensus_queue.cc:260] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:52.704540 15150 heartbeater.cc:503] Master 127.0.98.62:37239 was elected leader, sending a full tablet report...
I20260620 04:32:52.704808 15042 ts_tablet_manager.cc:1428] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 04:32:52.705294 15163 raft_consensus.cc:348] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:52.705543 15163 raft_consensus.cc:374] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:32:52.705593 15163 raft_consensus.cc:729] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb2561b7233497aa9a3680a7dc196c8, State: Initialized, Role: FOLLOWER
I20260620 04:32:52.705987 15163 consensus_queue.cc:260] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:52.706579 15017 heartbeater.cc:503] Master 127.0.98.62:37239 was elected leader, sending a full tablet report...
I20260620 04:32:52.706916 15163 ts_tablet_manager.cc:1428] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:32:52.743564 15167 raft_consensus.cc:482] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 04:32:52.743845 14880 tablet.cc:2369] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:32:52.743810 15167 raft_consensus.cc:504] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:52.745064 15167 leader_election.cc:288] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087), d37fdc9ed89d471e9426c13b2af2f87b (127.0.98.1:40883)
I20260620 04:32:52.748569 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.745285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46087 (local address 127.0.98.2:37001)
0620 04:32:52.745405 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:52.745418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:52.745514 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:52.745756 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:52.745759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:52.745778 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:52.745886 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.745891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.746599 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.746603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.747471 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.747480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.748155 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.748158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.748267 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.748285 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:32:52.748353 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:32:52.748424 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1}
I20260620 04:32:52.748567 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.745414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37001 (local address 127.0.98.3:46087)
0620 04:32:52.745540 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.745545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.745558 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:32:52.745623 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.745626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.745670 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.745734 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:52.745987 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.746494 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.747595 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.748036 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.748135 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.748204 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.748385 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.748453 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:32:52.749253 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.745705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39999 (local address 127.0.98.1:40883)
0620 04:32:52.745833 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.745837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.745985 (+ 148us) server_negotiation.cc:407] Connection header received
0620 04:32:52.746087 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.746090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.746141 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.746229 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:32:52.746475 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.747021 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.748124 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.748579 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.748678 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.748846 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.749090 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.749159 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:32:52.749284 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.745553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40883 (local address 127.0.98.2:39999)
0620 04:32:52.745909 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:52.745919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:52.746006 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:52.746252 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:52.746255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:52.746270 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:32:52.746377 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.746382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.747127 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.747130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.748001 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:52.748010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.748826 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.748830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:52.748923 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.748946 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:32:52.749066 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 04:32:52.749134 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":19,"thread_start_us":124,"threads_started":1}
I20260620 04:32:52.749285 15104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc6496e9c154c27882b226ee2d53316" candidate_uuid: "ceb2561b7233497aa9a3680a7dc196c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" is_pre_election: true
I20260620 04:32:52.749660 15104 raft_consensus.cc:2454] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb2561b7233497aa9a3680a7dc196c8 in term 0.
I20260620 04:32:52.749975 14833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc6496e9c154c27882b226ee2d53316" candidate_uuid: "ceb2561b7233497aa9a3680a7dc196c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" is_pre_election: true
I20260620 04:32:52.750161 14905 leader_election.cc:302] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28457bdb3b294aa49f6ebf41f837b8eb, ceb2561b7233497aa9a3680a7dc196c8; no voters:
I20260620 04:32:52.750309 14833 raft_consensus.cc:2454] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb2561b7233497aa9a3680a7dc196c8 in term 0.
I20260620 04:32:52.750500 15167 raft_consensus.cc:2781] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:32:52.750612 15167 raft_consensus.cc:482] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:32:52.750669 15167 raft_consensus.cc:3037] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:52.751732 15167 raft_consensus.cc:504] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:52.752041 15167 leader_election.cc:288] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [CANDIDATE]: Term 1 election: Requested vote from peers 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087), d37fdc9ed89d471e9426c13b2af2f87b (127.0.98.1:40883)
I20260620 04:32:52.752396 15104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc6496e9c154c27882b226ee2d53316" candidate_uuid: "ceb2561b7233497aa9a3680a7dc196c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28457bdb3b294aa49f6ebf41f837b8eb"
I20260620 04:32:52.752461 14833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc6496e9c154c27882b226ee2d53316" candidate_uuid: "ceb2561b7233497aa9a3680a7dc196c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37fdc9ed89d471e9426c13b2af2f87b"
I20260620 04:32:52.752502 15104 raft_consensus.cc:3037] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:52.752595 14833 raft_consensus.cc:3037] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:32:52.753526 15104 raft_consensus.cc:2454] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb2561b7233497aa9a3680a7dc196c8 in term 1.
I20260620 04:32:52.753693 14833 raft_consensus.cc:2454] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb2561b7233497aa9a3680a7dc196c8 in term 1.
I20260620 04:32:52.753800 14905 leader_election.cc:302] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28457bdb3b294aa49f6ebf41f837b8eb, ceb2561b7233497aa9a3680a7dc196c8; no voters:
I20260620 04:32:52.753957 15167 raft_consensus.cc:2781] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:32:52.754190 15167 raft_consensus.cc:686] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 1 LEADER]: Becoming Leader. State: Replica: ceb2561b7233497aa9a3680a7dc196c8, State: Running, Role: LEADER
I20260620 04:32:52.754501 15167 consensus_queue.cc:237] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:52.757742 14690 catalog_manager.cc:5495] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ceb2561b7233497aa9a3680a7dc196c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } health_report { overall_health: UNKNOWN } } }
I20260620 04:32:52.786209 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.782776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.0.98.2:36363)
0620 04:32:52.782900 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:52.782904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:52.782916 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:32:52.783039 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:52.783041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:52.783095 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:52.783191 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:32:52.783368 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.783867 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.784619 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:52.784866 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:52.784945 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:52.785345 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:32:52.785803 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:32:52.785897 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:52.786005 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 04:32:52.786060 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:32:52.792212 14833 raft_consensus.cc:1261] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Refusing update from remote peer ceb2561b7233497aa9a3680a7dc196c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:32:52.792335 15104 raft_consensus.cc:1261] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb [term 1 FOLLOWER]: Refusing update from remote peer ceb2561b7233497aa9a3680a7dc196c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:32:52.792876 15167 consensus_queue.cc:1035] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:32:52.793102 15169 consensus_queue.cc:1035] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:32:52.799389 15173 mvcc.cc:205] Tried to move back new op lower bound from 7298785168550285312 to 7298785168405188608. Current Snapshot: MvccSnapshot[applied={T|T < 7298785168550285312}]
I20260620 04:32:52.801127 15174 mvcc.cc:205] Tried to move back new op lower bound from 7298785168550285312 to 7298785168405188608. Current Snapshot: MvccSnapshot[applied={T|T < 7298785168550285312}]
I20260620 04:32:52.808189 14690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:32:52.808358 14690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:32:52.810818 14690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Sending DeleteTablet for 3 replicas of tablet afc6496e9c154c27882b226ee2d53316
I20260620 04:32:52.811623 14951 tablet_service.cc:1514] Processing DeleteTablet for tablet afc6496e9c154c27882b226ee2d53316 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:32:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33738
I20260620 04:32:52.811659 14813 tablet_service.cc:1514] Processing DeleteTablet for tablet afc6496e9c154c27882b226ee2d53316 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:32:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56826
I20260620 04:32:52.811700 15084 tablet_service.cc:1514] Processing DeleteTablet for tablet afc6496e9c154c27882b226ee2d53316 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:32:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37958
I20260620 04:32:52.812110 15182 tablet_replica.cc:330] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: stopping tablet replica
I20260620 04:32:52.812160 15183 tablet_replica.cc:330] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: stopping tablet replica
I20260620 04:32:52.812358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 14729
I20260620 04:32:52.812534 15182 raft_consensus.cc:2229] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:32:52.812495 15183 raft_consensus.cc:2229] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:32:52.812958 15183 raft_consensus.cc:2258] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:32:52.813158 15182 raft_consensus.cc:2258] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:32:52.814533 15183 ts_tablet_manager.cc:1905] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:32:52.814550 15182 ts_tablet_manager.cc:1905] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:32:52.817148 15183 ts_tablet_manager.cc:1918] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:32:52.817214 15182 ts_tablet_manager.cc:1918] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:32:52.817312 15182 log.cc:1196] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/afc6496e9c154c27882b226ee2d53316
I20260620 04:32:52.817328 15183 log.cc:1196] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/afc6496e9c154c27882b226ee2d53316
I20260620 04:32:52.817641 15182 ts_tablet_manager.cc:1939] T afc6496e9c154c27882b226ee2d53316 P ceb2561b7233497aa9a3680a7dc196c8: Deleting consensus metadata
I20260620 04:32:52.817739 15183 ts_tablet_manager.cc:1939] T afc6496e9c154c27882b226ee2d53316 P 28457bdb3b294aa49f6ebf41f837b8eb: Deleting consensus metadata
I20260620 04:32:52.818842 14677 catalog_manager.cc:4841] TS ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363): tablet afc6496e9c154c27882b226ee2d53316 (table test-table [id=9bab6d6ddacd4238a74ca20d5d42c0ad]) successfully deleted
I20260620 04:32:52.819118 14676 catalog_manager.cc:4841] TS 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087): tablet afc6496e9c154c27882b226ee2d53316 (table test-table [id=9bab6d6ddacd4238a74ca20d5d42c0ad]) successfully deleted
I20260620 04:32:52.820758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 14883
W20260620 04:32:52.820832 14676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40883 (error 108)
I20260620 04:32:52.825754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 15020
I20260620 04:32:52.825874 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:52.825541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40883 (local address 127.0.0.1:56832)
0620 04:32:52.825696 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:52.825766 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 04:32:52.826409 14676 catalog_manager.cc:4584] TS d37fdc9ed89d471e9426c13b2af2f87b (127.0.98.1:40883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afc6496e9c154c27882b226ee2d53316: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40883: connect: Connection refused (error 111)
I20260620 04:32:52.830708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 14658
I20260620 04:32:52.836127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37239
--webserver_interface=127.0.98.62
--webserver_port=35633
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:32:52.944392 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:52.944679 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:52.944720 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:52.948792 15185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:32:52.948882 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:52.948901 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:52.948916 15185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:32:52.948930 15185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:32:52.954425 15185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:52.955488 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:52.956523 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:32:52.965212 15185 server_base.cc:1034] running on GCE node
W20260620 04:32:52.965158 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:52.965143 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:52.965166 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:52.965926 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:52.967087 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:52.968281 15185 hybrid_clock.cc:648] HybridClock initialized: now 1781929972968244 us; error 70 us; skew 500 ppm
Jun 20 04:32:52 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:52.971248 15185 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:32:52.972211 15185 webserver.cc:460] Webserver started at http://127.0.98.62:35633/ using document root <none> and password file <none>
I20260620 04:32:52.972756 15185 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:52.972818 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:52.976838 15185 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:32:52.979241 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:52.980650 15185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:32:52.980785 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd"
format_stamp: "Formatted at 2026-06-20 04:32:50 on dist-test-slave-rzqm"
server_key: "4306e1b93d47cfd5f974198ff2aa21a8"
server_key_iv: "e78a57fe1bf7fd1903a8ebaf260f59d4"
server_key_version: "encryptionkey@0"
I20260620 04:32:52.981424 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:52.993362 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:52.996507 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:52.996727 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:53.003595 15185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37239
I20260620 04:32:53.003901 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37239 every 8 connection(s)
I20260620 04:32:53.004683 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:32:53.009054 15253 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:32:53.010699 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Bootstrap starting.
I20260620 04:32:53.012185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 15185
I20260620 04:32:53.012583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40883
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37239
--webserver_port=41843
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:32:53.027714 15253 log.cc:824] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:53.039117 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:32:53.039479 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd: Bootstrap complete.
I20260620 04:32:53.043340 15253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:53.043756 15253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1e7832e2e6456381c07c6f8e3f8fbd, State: Initialized, Role: FOLLOWER
I20260620 04:32:53.044247 15253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:53.044385 15253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:32:53.044456 15253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:32:53.044556 15253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 2 FOLLOWER]: Advancing to term 3
I20260620 04:32:53.046484 15253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:53.046809 15253 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f1e7832e2e6456381c07c6f8e3f8fbd; no voters:
I20260620 04:32:53.047127 15253 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 04:32:53.047241 15257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 3 FOLLOWER]: Leader election won for term 3
I20260620 04:32:53.047432 15257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [term 3 LEADER]: Becoming Leader. State: Replica: 7f1e7832e2e6456381c07c6f8e3f8fbd, State: Running, Role: LEADER
I20260620 04:32:53.047729 15257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } }
I20260620 04:32:53.048394 15253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:32:53.048703 15257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1e7832e2e6456381c07c6f8e3f8fbd. Latest consensus state: current_term: 3 leader_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } } }
I20260620 04:32:53.048915 15257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:53.050017 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1e7832e2e6456381c07c6f8e3f8fbd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37239 } } }
I20260620 04:32:53.050184 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1e7832e2e6456381c07c6f8e3f8fbd [sys.catalog]: This master's current role is: LEADER
I20260620 04:32:53.050377 15261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:32:53.054602 15261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:32:53.055166 15261 catalog_manager.cc:1259] Loaded cluster ID: 7d74f7cca0c54952ac60e92c98183965
I20260620 04:32:53.055226 15261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 04:32:53.056038 15261 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff5f529c9ad google::LogMessage::Fail() at ??:0
@ 0x7ff5f529f88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff5f529c4de google::LogMessage::Flush() at ??:0
@ 0x7ff5f529da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff5ffbdbfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff5ffbdcd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff5ffbd9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff5ffc1d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff5f74ee3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff5f74eec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff5f74f03f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff5f74df512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff5f8b236db start_thread at ??:0
@ 0x7ff5f3d9a61f clone at ??:0
W20260620 04:32:53.152968 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:53.153421 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:53.153491 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:53.159853 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:53.159948 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:53.160041 15255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:32:53.168370 15255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41843
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:53.169914 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:53.171224 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:53.182094 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:53.182389 15255 server_base.cc:1034] running on GCE node
W20260620 04:32:53.182087 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:53.182089 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:53.182951 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:53.183636 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:53.184830 15255 hybrid_clock.cc:648] HybridClock initialized: now 1781929973184792 us; error 56 us; skew 500 ppm
Jun 20 04:32:53 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:53.187748 15255 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:32:53.188740 15255 webserver.cc:460] Webserver started at http://127.0.98.1:41843/ using document root <none> and password file <none>
I20260620 04:32:53.189342 15255 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:53.189414 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:53.193485 15255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:32:53.195778 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:53.197206 15255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:32:53.197319 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "d37fdc9ed89d471e9426c13b2af2f87b"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "9b9ffc081708367ae8d41f01adfeec82"
server_key_iv: "a4c53f0626a6e4146933654de0a9df28"
server_key_version: "encryptionkey@0"
I20260620 04:32:53.197876 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:53.218598 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:53.221793 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:53.221977 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:53.222594 15255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:53.223716 15311 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:32:53.229359 15255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:32:53.229453 15255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 04:32:53.229537 15255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:32:53.231339 15255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:32:53.231395 15255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 04:32:53.231737 15311 tablet_bootstrap.cc:492] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Bootstrap starting.
I20260620 04:32:53.241185 15255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40883
I20260620 04:32:53.241462 15418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40883 every 8 connection(s)
I20260620 04:32:53.242245 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:32:53.249644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 15255
I20260620 04:32:53.250051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36363
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37239
--webserver_port=44585
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:32:53.255271 15311 log.cc:824] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:32:53.263000 15311 tablet_bootstrap.cc:492] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:32:53.263367 15311 tablet_bootstrap.cc:492] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Bootstrap complete.
I20260620 04:32:53.264302 15311 ts_tablet_manager.cc:1397] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.008s
I20260620 04:32:53.266794 15311 raft_consensus.cc:348] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:53.267221 15311 raft_consensus.cc:729] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37fdc9ed89d471e9426c13b2af2f87b, State: Initialized, Role: FOLLOWER
I20260620 04:32:53.267690 15311 consensus_queue.cc:260] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:53.268550 15311 ts_tablet_manager.cc:1428] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260620 04:32:53.311471 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:32:53.244226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:42493)
0620 04:32:53.244728 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:53.244784 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:53.245647 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:53.310742 (+ 65095us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
W20260620 04:32:53.311964 15419 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37239 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37239: Client connection negotiation failed: client connection to 127.0.98.62:37239: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:32:53.312665 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:53.312415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:56147)
0620 04:32:53.312532 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:53.312602 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 04:32:53.313318 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:53.313119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:54115)
0620 04:32:53.313218 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:53.313265 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 04:32:53.313463 15419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:32:53.381672 15422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:53.381948 15422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:53.382010 15422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:53.386075 15422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:53.386147 15422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:53.386229 15422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:32:53.391896 15422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44585
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:53.393064 15422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:53.394148 15422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:53.403911 15432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:53.403898 15429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:53.403898 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:53.404162 15422 server_base.cc:1034] running on GCE node
I20260620 04:32:53.404754 15422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:53.405468 15422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:53.406688 15422 hybrid_clock.cc:648] HybridClock initialized: now 1781929973406668 us; error 36 us; skew 500 ppm
Jun 20 04:32:53 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:53.409524 15422 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:32:53.410440 15422 webserver.cc:460] Webserver started at http://127.0.98.2:44585/ using document root <none> and password file <none>
I20260620 04:32:53.410980 15422 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:53.411046 15422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:53.414856 15422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:32:53.416980 15439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:53.418299 15422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:32:53.418421 15422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "ceb2561b7233497aa9a3680a7dc196c8"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "6f7d60c48624067841cbfe62b1c44aa8"
server_key_iv: "3ded53bf6e455a24ea98e38f1c95badd"
server_key_version: "encryptionkey@0"
I20260620 04:32:53.418979 15422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:53.429093 15422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:53.432147 15422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:53.432333 15422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:53.432917 15422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:53.433763 15422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:53.433808 15422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:53.433861 15422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:53.433889 15422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:53.442520 15422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36363
I20260620 04:32:53.442548 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36363 every 8 connection(s)
I20260620 04:32:53.443506 15422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:32:53.447186 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:53.445588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:60101)
0620 04:32:53.446341 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:53.446457 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":104,"threads_started":1}
W20260620 04:32:53.447640 15553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37239 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37239: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
I20260620 04:32:53.448016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 15422
I20260620 04:32:53.448359 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:53.448100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:54039)
0620 04:32:53.448234 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:53.448291 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 04:32:53.448451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46087
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37239
--webserver_port=35129
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:32:53.449060 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:53.448804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:49285)
0620 04:32:53.448926 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:53.448973 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 04:32:53.449256 15553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:32:53.556965 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:32:53.557274 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:32:53.557339 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:32:53.561429 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:32:53.561501 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:32:53.561587 15556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:32:53.567021 15556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35129
--tserver_master_addrs=127.0.98.62:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:32:53.568190 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:32:53.569258 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:32:53.578714 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:32:53.578831 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:53.578873 15556 server_base.cc:1034] running on GCE node
W20260620 04:32:53.578714 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:32:53.579413 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:32:53.580046 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:32:53.581199 15556 hybrid_clock.cc:648] HybridClock initialized: now 1781929973581168 us; error 41 us; skew 500 ppm
Jun 20 04:32:53 dist-test-slave-rzqm krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:32:53.583819 15556 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:32:53.584698 15556 webserver.cc:460] Webserver started at http://127.0.98.3:35129/ using document root <none> and password file <none>
I20260620 04:32:53.585258 15556 fs_manager.cc:359] Metadata directory not provided
I20260620 04:32:53.585321 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:32:53.589082 15556 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 04:32:53.591060 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:53.592275 15556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:32:53.592379 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "28457bdb3b294aa49f6ebf41f837b8eb"
format_stamp: "Formatted at 2026-06-20 04:32:51 on dist-test-slave-rzqm"
server_key: "9e152fd15ff1c83165c1bd0d531567bb"
server_key_iv: "24ab378683c81cd9dd8693d659943d84"
server_key_version: "encryptionkey@0"
I20260620 04:32:53.592926 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:32:53.616637 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:32:53.619843 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:32:53.620039 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:32:53.620811 15556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:32:53.621784 15556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:32:53.621835 15556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:53.621874 15556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:32:53.621888 15556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:32:53.630995 15556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46087
I20260620 04:32:53.631079 15684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46087 every 8 connection(s)
I20260620 04:32:53.632011 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:32:53.635046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 15556
I20260620 04:32:53.636102 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:53.634201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:51919)
0620 04:32:53.634755 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:53.635284 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":114,"threads_started":1}
W20260620 04:32:53.636521 15685 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37239 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37239: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
I20260620 04:32:53.637100 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:53.636853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:46485)
0620 04:32:53.636957 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:53.637037 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 04:32:53.637739 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:53.637552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:54613)
0620 04:32:53.637647 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:53.637690 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 04:32:53.637889 15685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:32:54.314662 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:54.314130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:33449)
0620 04:32:54.314452 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:54.314515 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":75,"threads_started":1}
I20260620 04:32:54.450433 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:54.449917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:35253)
0620 04:32:54.450225 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:54.450294 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":81,"threads_started":1}
I20260620 04:32:54.638880 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:54.638458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:49919)
0620 04:32:54.638726 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:54.638781 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":74,"threads_started":1}
I20260620 04:32:54.898162 15692 raft_consensus.cc:482] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:32:54.898379 15692 raft_consensus.cc:504] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:54.899523 15692 leader_election.cc:288] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363), 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087)
Jun 20 04:32:54 dist-test-slave-rzqm krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 04:32:54 dist-test-slave-rzqm krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 04:32:54.912837 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:54.899978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33765 (local address 127.0.98.3:46087)
0620 04:32:54.900343 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:54.900365 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:54.900401 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:32:54.901167 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:54.901182 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:54.901533 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:54.901779 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:54.903200 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:54.904152 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:54.905434 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:54.905886 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:54.906033 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:54.908602 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:54.908607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:54.908619 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:54.908648 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:54.910402 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:54.910978 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:54.910983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:54.910987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:54.911095 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:54.911400 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:54.911403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:54.911405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:54.911800 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:54.911910 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:54.912409 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 04:32:54.912648 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":69,"threads_started":1}
I20260620 04:32:54.912833 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:54.899782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46087 (local address 127.0.98.1:33765)
0620 04:32:54.900224 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:54.900237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:54.900373 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:54.901881 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:54.901887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:54.902499 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:54.903034 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:54.903055 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:54.904290 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:54.904293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:54.905310 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:54.905320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:54.905993 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:54.905996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:54.906236 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:54.906846 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:54.906857 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:54.908395 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:54.910513 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:54.910519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:54.910532 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:54.910875 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:54.911177 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:54.911181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:54.911183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:54.911319 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:54.911901 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:54.911904 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:54.912130 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 04:32:54.912289 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 04:32:54.912626 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":198,"spinlock_wait_cycles":24576,"thread_start_us":70,"threads_started":1}
I20260620 04:32:54.912837 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:54.899949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56669 (local address 127.0.98.2:36363)
0620 04:32:54.900624 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 04:32:54.900644 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:32:54.900682 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:32:54.901368 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:32:54.901375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:32:54.901684 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:32:54.902007 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:32:54.903198 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:54.904169 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:54.905397 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:32:54.905871 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:32:54.906019 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:54.908859 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:32:54.908864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:32:54.908874 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:32:54.908902 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:32:54.910480 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:54.910983 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:54.910988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:54.910992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:54.911094 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:32:54.911407 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:32:54.911410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:32:54.911412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:32:54.911789 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:32:54.911905 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:32:54.912410 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 04:32:54.912652 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":68,"threads_started":1}
I20260620 04:32:54.912879 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:54.899782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36363 (local address 127.0.98.1:56669)
0620 04:32:54.900109 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:54.900124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:32:54.900243 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:32:54.901965 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:32:54.901968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:32:54.902501 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:32:54.903033 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:54.903056 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:54.904295 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:54.904299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:54.905288 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:32:54.905298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:32:54.905984 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:32:54.905987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:32:54.906233 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:32:54.906800 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:32:54.906816 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:32:54.908697 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:32:54.910585 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:54.910588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:54.910590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:54.910885 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:54.911184 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:32:54.911187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:32:54.911189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:32:54.911326 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:32:54.911896 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:32:54.911902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:32:54.912130 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 04:32:54.912289 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 04:32:54.912707 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":268,"spinlock_wait_cycles":12032,"thread_start_us":74,"threads_started":1}
W20260620 04:32:54.915778 15307 leader_election.cc:341] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363): Not found: Tablet not found: afc6496e9c154c27882b226ee2d53316
W20260620 04:32:54.916035 15306 leader_election.cc:341] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087): Not found: Tablet not found: afc6496e9c154c27882b226ee2d53316
I20260620 04:32:54.916139 15306 leader_election.cc:302] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d37fdc9ed89d471e9426c13b2af2f87b; no voters: 28457bdb3b294aa49f6ebf41f837b8eb, ceb2561b7233497aa9a3680a7dc196c8
I20260620 04:32:54.916404 15692 raft_consensus.cc:2726] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:32:55.315851 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:55.315506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:45083)
0620 04:32:55.315699 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:55.315765 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 04:32:55.451786 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:55.451328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:47733)
0620 04:32:55.451617 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:55.451684 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260620 04:32:55.640071 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:55.639653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:56811)
0620 04:32:55.639917 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:55.639969 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
I20260620 04:32:56.317265 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:56.316718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:55237)
0620 04:32:56.317094 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:56.317161 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260620 04:32:56.453229 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:56.452723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:47497)
0620 04:32:56.453027 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:56.453098 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
I20260620 04:32:56.530112 15701 raft_consensus.cc:482] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:32:56.530259 15701 raft_consensus.cc:504] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:56.530678 15701 leader_election.cc:288] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363), 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087)
W20260620 04:32:56.531448 15306 leader_election.cc:341] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087): Not found: Tablet not found: afc6496e9c154c27882b226ee2d53316
W20260620 04:32:56.531558 15307 leader_election.cc:341] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363): Not found: Tablet not found: afc6496e9c154c27882b226ee2d53316
I20260620 04:32:56.531656 15307 leader_election.cc:302] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d37fdc9ed89d471e9426c13b2af2f87b; no voters: 28457bdb3b294aa49f6ebf41f837b8eb, ceb2561b7233497aa9a3680a7dc196c8
I20260620 04:32:56.531803 15701 raft_consensus.cc:2726] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:32:56.641112 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:56.640732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:54261)
0620 04:32:56.640938 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:56.640990 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260620 04:32:57.318498 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:57.318046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:36037)
0620 04:32:57.318335 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:57.318402 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260620 04:32:57.454489 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:57.454032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:47249)
0620 04:32:57.454323 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:57.454393 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260620 04:32:57.642452 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:57.641946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:44057)
0620 04:32:57.642264 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:57.642339 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":96,"threads_started":1}
I20260620 04:32:58.319991 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:58.319511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:39987)
0620 04:32:58.319826 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:58.319892 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":76,"threads_started":1}
I20260620 04:32:58.455740 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:58.455314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:34977)
0620 04:32:58.455566 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:58.455637 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260620 04:32:58.643612 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:58.643221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:40341)
0620 04:32:58.643461 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:58.643518 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
I20260620 04:32:58.841270 15709 raft_consensus.cc:482] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:32:58.841418 15709 raft_consensus.cc:504] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:32:58.841820 15709 leader_election.cc:288] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363), 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087)
W20260620 04:32:58.842694 15306 leader_election.cc:341] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087): Not found: Tablet not found: afc6496e9c154c27882b226ee2d53316
W20260620 04:32:58.842816 15307 leader_election.cc:341] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363): Not found: Tablet not found: afc6496e9c154c27882b226ee2d53316
I20260620 04:32:58.842876 15307 leader_election.cc:302] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d37fdc9ed89d471e9426c13b2af2f87b; no voters: 28457bdb3b294aa49f6ebf41f837b8eb, ceb2561b7233497aa9a3680a7dc196c8
I20260620 04:32:58.843021 15709 raft_consensus.cc:2726] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:32:59.321336 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:59.320854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:58713)
0620 04:32:59.321145 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:59.321238 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260620 04:32:59.457070 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:59.456664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:48235)
0620 04:32:59.456903 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:59.456956 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
I20260620 04:32:59.644737 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:32:59.644326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:41999)
0620 04:32:59.644585 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 04:32:59.644642 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
I20260620 04:33:00.322702 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:00.322151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:59005)
0620 04:33:00.322494 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:00.322597 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260620 04:33:00.458292 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:00.457813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:58653)
0620 04:33:00.458132 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:00.458195 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":74,"threads_started":1}
I20260620 04:33:00.645980 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:00.645529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:36813)
0620 04:33:00.645817 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:00.645879 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
I20260620 04:33:01.234993 15716 raft_consensus.cc:482] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:33:01.235153 15716 raft_consensus.cc:504] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:33:01.235577 15716 leader_election.cc:288] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363), 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087)
W20260620 04:33:01.236505 15306 leader_election.cc:341] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087): Not found: Tablet not found: afc6496e9c154c27882b226ee2d53316
W20260620 04:33:01.236649 15307 leader_election.cc:341] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363): Not found: Tablet not found: afc6496e9c154c27882b226ee2d53316
I20260620 04:33:01.236747 15307 leader_election.cc:302] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d37fdc9ed89d471e9426c13b2af2f87b; no voters: 28457bdb3b294aa49f6ebf41f837b8eb, ceb2561b7233497aa9a3680a7dc196c8
I20260620 04:33:01.236896 15716 raft_consensus.cc:2726] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:33:01.324090 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:01.323623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:58211)
0620 04:33:01.323926 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:01.323993 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
I20260620 04:33:01.459609 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:01.459179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:58871)
0620 04:33:01.459446 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:01.459508 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260620 04:33:01.647226 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:01.646782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:50695)
0620 04:33:01.647069 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:01.647130 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260620 04:33:02.325395 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:02.324928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:34843)
0620 04:33:02.325219 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:02.325274 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
I20260620 04:33:02.460887 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:02.460428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:43825)
0620 04:33:02.460726 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:02.460790 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260620 04:33:02.648425 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:02.647966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:59947)
0620 04:33:02.648263 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:02.648328 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
I20260620 04:33:03.326661 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:03.326238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:58749)
0620 04:33:03.326506 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:03.326566 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":87,"threads_started":1}
I20260620 04:33:03.462172 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:03.461713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:35937)
0620 04:33:03.461999 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:03.462066 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260620 04:33:03.649825 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:03.649321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:40067)
0620 04:33:03.649636 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:03.649717 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260620 04:33:04.327872 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:04.327406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:41665)
0620 04:33:04.327693 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:04.327761 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260620 04:33:04.463435 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:04.463000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:55151)
0620 04:33:04.463271 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:04.463337 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260620 04:33:04.651053 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:04.650646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:57489)
0620 04:33:04.650895 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:04.650958 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260620 04:33:05.329253 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:05.328753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:54665)
0620 04:33:05.329077 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:05.329141 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
I20260620 04:33:05.464666 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:05.464182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:55567)
0620 04:33:05.464481 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:05.464555 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
I20260620 04:33:05.652302 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:05.651817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:34059)
0620 04:33:05.652125 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:05.652191 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":80,"threads_started":1}
I20260620 04:33:06.166078 15732 raft_consensus.cc:482] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:33:06.166217 15732 raft_consensus.cc:504] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb2561b7233497aa9a3680a7dc196c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36363 } } peers { permanent_uuid: "28457bdb3b294aa49f6ebf41f837b8eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46087 } } peers { permanent_uuid: "d37fdc9ed89d471e9426c13b2af2f87b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40883 } }
I20260620 04:33:06.166671 15732 leader_election.cc:288] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363), 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087)
W20260620 04:33:06.167469 15306 leader_election.cc:341] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28457bdb3b294aa49f6ebf41f837b8eb (127.0.98.3:46087): Not found: Tablet not found: afc6496e9c154c27882b226ee2d53316
W20260620 04:33:06.167570 15307 leader_election.cc:341] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ceb2561b7233497aa9a3680a7dc196c8 (127.0.98.2:36363): Not found: Tablet not found: afc6496e9c154c27882b226ee2d53316
I20260620 04:33:06.167670 15307 leader_election.cc:302] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d37fdc9ed89d471e9426c13b2af2f87b; no voters: 28457bdb3b294aa49f6ebf41f837b8eb, ceb2561b7233497aa9a3680a7dc196c8
I20260620 04:33:06.167830 15732 raft_consensus.cc:2726] T afc6496e9c154c27882b226ee2d53316 P d37fdc9ed89d471e9426c13b2af2f87b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:33:06.330670 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:06.330110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:55653)
0620 04:33:06.330504 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:06.330572 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":90,"threads_started":1}
I20260620 04:33:06.466080 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:06.465619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:44593)
0620 04:33:06.465911 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:06.465965 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260620 04:33:06.653571 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:06.653139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:43141)
0620 04:33:06.653421 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:06.653479 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
I20260620 04:33:07.331933 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:07.331511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:34799)
0620 04:33:07.331763 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:07.331829 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260620 04:33:07.467212 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:07.466766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:51389)
0620 04:33:07.467061 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:07.467117 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
I20260620 04:33:07.654760 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:07.654346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.3:32967)
0620 04:33:07.654603 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:07.654663 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260620 04:33:08.333318 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:08.332881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.1:48895)
0620 04:33:08.333145 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:08.333198 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260620 04:33:08.468525 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:08.468072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37239 (local address 127.0.98.2:57833)
0620 04:33:08.468357 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:08.468420 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
I20260620 04:33:08.639629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 15255
I20260620 04:33:08.644752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 15422
I20260620 04:33:08.649546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 15556
2026-06-20T04:33:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19112 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:33:08 dist-test-slave-rzqm krb5kdc[15743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:33:08 dist-test-slave-rzqm krb5kdc[15743](info): set up 2 sockets
Jun 20 04:33:08 dist-test-slave-rzqm krb5kdc[15743](info): commencing operation
krb5kdc: starting...
W20260620 04:33:09.601293 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:33:09 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:33:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:33:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:33:09.720112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41243
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:33:09.828886 15756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:09.829178 15756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:09.829219 15756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:09.833192 15756 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 04:33:09.833245 15756 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 04:33:09.833259 15756 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 04:33:09.833271 15756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:33:09.833288 15756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:09.833304 15756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:09.833318 15756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:33:09.833333 15756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:33:09.838629 15756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:09.839620 15756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:09.840602 15756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:09.849324 15761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:09.849373 15756 server_base.cc:1034] running on GCE node
W20260620 04:33:09.849493 15764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:09.849385 15762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:09.850157 15756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:09.851281 15756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:09.852458 15756 hybrid_clock.cc:648] HybridClock initialized: now 1781929989852443 us; error 38 us; skew 500 ppm
Jun 20 04:33:09 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:09.855209 15756 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:33:09.856217 15756 webserver.cc:460] Webserver started at http://127.0.98.62:45615/ using document root <none> and password file <none>
I20260620 04:33:09.856813 15756 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:09.856880 15756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:09.857131 15756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:33:09.859290 15756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "a658e635bae74110a620ebba15b66a15"
format_stamp: "Formatted at 2026-06-20 04:33:09 on dist-test-slave-rzqm"
server_key: "531fc9d51146ce5fb2c861f8273d956f"
server_key_iv: "dd94649c9fec9e099a7bab8b0276ef63"
server_key_version: "encryptionkey@0"
I20260620 04:33:09.859839 15756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "a658e635bae74110a620ebba15b66a15"
format_stamp: "Formatted at 2026-06-20 04:33:09 on dist-test-slave-rzqm"
server_key: "531fc9d51146ce5fb2c861f8273d956f"
server_key_iv: "dd94649c9fec9e099a7bab8b0276ef63"
server_key_version: "encryptionkey@0"
I20260620 04:33:09.863694 15756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:33:09.866276 15771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:09.867645 15756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 04:33:09.867782 15756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "a658e635bae74110a620ebba15b66a15"
format_stamp: "Formatted at 2026-06-20 04:33:09 on dist-test-slave-rzqm"
server_key: "531fc9d51146ce5fb2c861f8273d956f"
server_key_iv: "dd94649c9fec9e099a7bab8b0276ef63"
server_key_version: "encryptionkey@0"
I20260620 04:33:09.867911 15756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:09.881770 15756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:09.885202 15756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:09.885376 15756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:09.892436 15756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41243
I20260620 04:33:09.892720 15823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41243 every 8 connection(s)
I20260620 04:33:09.893483 15756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:33:09.896282 15824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:33:09.896339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 15756
I20260620 04:33:09.896441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:33:09.896765 392 external_mini_cluster.cc:1428] Setting key 7935e3ff3b6ce47598e24bd20d17bf45
I20260620 04:33:09.903548 15824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Bootstrap starting.
I20260620 04:33:09.906220 15824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Neither blocks nor log segments found. Creating new log.
Jun 20 04:33:09 dist-test-slave-rzqm krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:33:09.907049 15824 log.cc:824] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Log is configured to *not* fsync() on all Append() calls
I20260620 04:33:09.909399 15824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: No bootstrap required, opened a new log
I20260620 04:33:09.911971 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:09.898366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.0.98.62:41243)
0620 04:33:09.898798 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:09.898817 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:09.898861 (+ 44us) server_negotiation.cc:407] Connection header received
0620 04:33:09.899635 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:09.899660 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:09.900117 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:09.900511 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:09.901244 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:09.902748 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:09.903600 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:09.904267 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:09.904415 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:09.906966 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:09.906974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:09.906991 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:09.907025 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:09.908884 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:09.909352 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:09.909357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:09.909363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:09.909505 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:09.909767 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:09.909770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:09.909772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:09.910287 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:09.910483 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:09.910682 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 04:33:09.910987 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260620 04:33:09.913568 15824 raft_consensus.cc:348] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:09.913784 15824 raft_consensus.cc:374] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:33:09.913816 15824 raft_consensus.cc:729] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a658e635bae74110a620ebba15b66a15, State: Initialized, Role: FOLLOWER
I20260620 04:33:09.914235 15824 consensus_queue.cc:260] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:09.914361 15824 raft_consensus.cc:388] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:33:09.914419 15824 raft_consensus.cc:482] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:33:09.914520 15824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:33:09.915385 15824 raft_consensus.cc:504] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:09.915709 15824 leader_election.cc:302] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a658e635bae74110a620ebba15b66a15; no voters:
I20260620 04:33:09.916000 15824 leader_election.cc:288] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:33:09.916100 15829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:33:09.916301 15829 raft_consensus.cc:686] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 1 LEADER]: Becoming Leader. State: Replica: a658e635bae74110a620ebba15b66a15, State: Running, Role: LEADER
I20260620 04:33:09.916633 15829 consensus_queue.cc:237] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:09.917198 15824 sys_catalog.cc:564] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:33:09.918233 15831 sys_catalog.cc:455] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a658e635bae74110a620ebba15b66a15. Latest consensus state: current_term: 1 leader_uuid: "a658e635bae74110a620ebba15b66a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } } }
I20260620 04:33:09.918360 15831 sys_catalog.cc:458] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: This master's current role is: LEADER
I20260620 04:33:09.918658 15830 sys_catalog.cc:455] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a658e635bae74110a620ebba15b66a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } } }
I20260620 04:33:09.918746 15838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:33:09.918776 15830 sys_catalog.cc:458] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: This master's current role is: LEADER
I20260620 04:33:09.922317 15838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:33:09.928148 15838 catalog_manager.cc:1347] Generated new cluster ID: 7249911cc34c407d936a2751d661332b
I20260620 04:33:09.928236 15838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:33:09.957242 15838 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:33:09.958252 15838 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:33:09.963110 15838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Generated new TSK 0
I20260620 04:33:09.963961 15838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:33:09.999559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41243
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:33:10.109603 15850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:10.109903 15850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:10.109956 15850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:10.114005 15850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:10.114073 15850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:10.114131 15850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:33:10.120057 15850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:10.121265 15850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:10.122354 15850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:10.132375 15856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:10.132385 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:10.132397 15855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:10.132622 15850 server_base.cc:1034] running on GCE node
I20260620 04:33:10.133220 15850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:10.133952 15850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:10.135121 15850 hybrid_clock.cc:648] HybridClock initialized: now 1781929990135100 us; error 34 us; skew 500 ppm
Jun 20 04:33:10 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:10.138037 15850 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:33:10.138970 15850 webserver.cc:460] Webserver started at http://127.0.98.1:45649/ using document root <none> and password file <none>
I20260620 04:33:10.139550 15850 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:10.139617 15850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:10.139804 15850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:33:10.141981 15850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "83c690c8b83242238ea732260f883fe4"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "8e3ba8dd39fd697b6619902a75527611"
server_key_iv: "c13ca6d386310f23a1a81f8327800e3f"
server_key_version: "encryptionkey@0"
I20260620 04:33:10.142535 15850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "83c690c8b83242238ea732260f883fe4"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "8e3ba8dd39fd697b6619902a75527611"
server_key_iv: "c13ca6d386310f23a1a81f8327800e3f"
server_key_version: "encryptionkey@0"
I20260620 04:33:10.146363 15850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:33:10.148860 15865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:10.150285 15850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:33:10.150405 15850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "83c690c8b83242238ea732260f883fe4"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "8e3ba8dd39fd697b6619902a75527611"
server_key_iv: "c13ca6d386310f23a1a81f8327800e3f"
server_key_version: "encryptionkey@0"
I20260620 04:33:10.150525 15850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:10.171591 15850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:10.174746 15850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:10.174907 15850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:10.175510 15850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:33:10.176312 15850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:33:10.176360 15850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:10.176399 15850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:33:10.176414 15850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:10.184785 15850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43213
I20260620 04:33:10.184868 15978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43213 every 8 connection(s)
I20260620 04:33:10.185719 15850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 04:33:10 dist-test-slave-rzqm krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:33:10.195952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 15850
I20260620 04:33:10.196086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:33:10.196421 392 external_mini_cluster.cc:1428] Setting key a41182f713d743514c33ba005f785c3b
I20260620 04:33:10.199999 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.187487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46549 (local address 127.0.98.62:41243)
0620 04:33:10.187612 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:10.187616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:10.188893 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 04:33:10.189872 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:10.189875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:10.189932 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:10.190006 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:10.191242 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.191786 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.192934 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.193224 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.193310 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.195720 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:10.195724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:10.195726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:10.195754 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:10.197309 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:10.197841 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:10.197845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:10.197847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:10.197959 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:10.198373 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:10.198377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:10.198379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:10.198630 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:10.198733 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:10.199550 (+ 817us) server_negotiation.cc:299] Negotiation successful
0620 04:33:10.199757 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:33:10.200938 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.187862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:46549)
0620 04:33:10.188739 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:10.188784 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:10.189605 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:10.190195 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:10.190203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:10.190600 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:33:10.191090 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.191103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.191912 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.191914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.192812 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.192822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.193383 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.193387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.193542 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.194098 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:33:10.194113 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:33:10.195560 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:33:10.197417 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:10.197423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:10.197439 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:10.197742 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:10.198060 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:10.198063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:10.198065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:10.198242 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:10.198791 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:33:10.198798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:33:10.199194 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 04:33:10.199480 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 04:33:10.199842 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":526,"threads_started":1}
I20260620 04:33:10.202385 15979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41243
I20260620 04:33:10.202673 15979 heartbeater.cc:464] Registering TS with master...
I20260620 04:33:10.203284 15979 heartbeater.cc:511] Master 127.0.98.62:41243 requested a full tablet report, sending...
I20260620 04:33:10.205155 15788 ts_manager.cc:194] Registered new tserver with Master: 83c690c8b83242238ea732260f883fe4 (127.0.98.1:43213)
I20260620 04:33:10.206784 15788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46549
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:33:10.224231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41243
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:33:10.334184 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:10.334450 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:10.334511 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:10.338591 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:10.338661 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:10.338742 15984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:33:10.344069 15984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:10.345203 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:10.346233 15984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:10.356045 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:10.356287 15984 server_base.cc:1034] running on GCE node
W20260620 04:33:10.356045 15989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:10.356171 15992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:10.356828 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:10.357602 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:10.358774 15984 hybrid_clock.cc:648] HybridClock initialized: now 1781929990358755 us; error 27 us; skew 500 ppm
Jun 20 04:33:10 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:10.361624 15984 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:33:10.362520 15984 webserver.cc:460] Webserver started at http://127.0.98.2:44269/ using document root <none> and password file <none>
I20260620 04:33:10.363054 15984 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:10.363111 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:10.363288 15984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:33:10.365465 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "cf115ee8455e4fb286d8799f4c9d469f"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "425fbf5ec548f30430539f883196c4ea"
server_key_iv: "cd61e4d3a859b556bbd79c613553d5bc"
server_key_version: "encryptionkey@0"
I20260620 04:33:10.366011 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cf115ee8455e4fb286d8799f4c9d469f"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "425fbf5ec548f30430539f883196c4ea"
server_key_iv: "cd61e4d3a859b556bbd79c613553d5bc"
server_key_version: "encryptionkey@0"
I20260620 04:33:10.370009 15984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:33:10.372651 15999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:10.374030 15984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:33:10.374154 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "cf115ee8455e4fb286d8799f4c9d469f"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "425fbf5ec548f30430539f883196c4ea"
server_key_iv: "cd61e4d3a859b556bbd79c613553d5bc"
server_key_version: "encryptionkey@0"
I20260620 04:33:10.374262 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:10.397639 15984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:10.400837 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:10.401037 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:10.401633 15984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:33:10.402465 15984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:33:10.402514 15984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:10.402566 15984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:33:10.402595 15984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:10.411414 15984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37125
I20260620 04:33:10.411458 16112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37125 every 8 connection(s)
I20260620 04:33:10.412307 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:33:10.420861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 15984
I20260620 04:33:10.420964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:33:10.421258 392 external_mini_cluster.cc:1428] Setting key 68759574ef62d92e1a79b5a21bbceec0
Jun 20 04:33:10 dist-test-slave-rzqm krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:33:10.425766 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.414086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59297 (local address 127.0.98.62:41243)
0620 04:33:10.414210 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:10.414213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:10.415215 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 04:33:10.416147 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:10.416151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:10.416204 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:10.416277 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:10.417540 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.418214 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.419388 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.419600 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.419684 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.422124 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:10.422127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:10.422129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:10.422154 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:10.423701 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:10.424216 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:10.424219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:10.424221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:10.424319 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:10.424666 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:10.424668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:10.424670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:10.424835 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:10.424906 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:10.425535 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 04:33:10.425646 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:33:10.426898 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.414456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:59297)
0620 04:33:10.415073 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:10.415113 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:10.415936 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:10.416462 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:10.416469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:10.416838 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:33:10.417407 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.417422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.418336 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.418339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.419286 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.419295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.419730 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.419734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.419891 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.420430 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:33:10.420447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:33:10.421954 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:33:10.423838 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:10.423843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:10.423855 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:10.424112 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:10.424402 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:10.424404 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:10.424406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:10.424587 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:10.424920 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:33:10.424926 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:33:10.425286 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 04:33:10.425502 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 04:33:10.425851 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":111,"threads_started":1}
I20260620 04:33:10.428128 16113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41243
I20260620 04:33:10.428378 16113 heartbeater.cc:464] Registering TS with master...
I20260620 04:33:10.428862 16113 heartbeater.cc:511] Master 127.0.98.62:41243 requested a full tablet report, sending...
I20260620 04:33:10.429958 15788 ts_manager.cc:194] Registered new tserver with Master: cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:33:10.430718 15788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:33:10.446200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41243
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:33:10.556843 16118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:10.557173 16118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:10.557222 16118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:10.561296 16118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:10.561355 16118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:10.561422 16118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:33:10.566759 16118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:10.567859 16118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:10.568888 16118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:10.578462 16126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:10.578462 16123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:10.578469 16124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:10.578843 16118 server_base.cc:1034] running on GCE node
I20260620 04:33:10.579234 16118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:10.579880 16118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:10.581051 16118 hybrid_clock.cc:648] HybridClock initialized: now 1781929990581034 us; error 33 us; skew 500 ppm
Jun 20 04:33:10 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:10.584702 16118 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:33:10.586081 16118 webserver.cc:460] Webserver started at http://127.0.98.3:43037/ using document root <none> and password file <none>
I20260620 04:33:10.586830 16118 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:10.586927 16118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:10.587193 16118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:33:10.590167 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d8aedc4954a47b5b50150798daea3ba"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "0eee212138622a97c926a6efe647dcce"
server_key_iv: "b790f90dc228b2d5333ae640d197716d"
server_key_version: "encryptionkey@0"
I20260620 04:33:10.590868 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d8aedc4954a47b5b50150798daea3ba"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "0eee212138622a97c926a6efe647dcce"
server_key_iv: "b790f90dc228b2d5333ae640d197716d"
server_key_version: "encryptionkey@0"
I20260620 04:33:10.595214 16118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:33:10.597749 16133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:10.599156 16118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:33:10.599278 16118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "6d8aedc4954a47b5b50150798daea3ba"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "0eee212138622a97c926a6efe647dcce"
server_key_iv: "b790f90dc228b2d5333ae640d197716d"
server_key_version: "encryptionkey@0"
I20260620 04:33:10.599400 16118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:10.628641 16118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:10.632141 16118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:10.632342 16118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:10.632963 16118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:33:10.633839 16118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:33:10.633889 16118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:10.633941 16118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:33:10.633970 16118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:10.642813 16118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46603
I20260620 04:33:10.642859 16246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46603 every 8 connection(s)
I20260620 04:33:10.643759 16118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:33:10.652884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 16118
I20260620 04:33:10.653045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
Jun 20 04:33:10 dist-test-slave-rzqm krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781929990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:33:10.653366 392 external_mini_cluster.cc:1428] Setting key 24c40b0b124800bde30c8cc5cc6df6e4
I20260620 04:33:10.657888 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.645733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33287 (local address 127.0.98.62:41243)
0620 04:33:10.645955 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:10.645959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:10.646984 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 04:33:10.647919 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:10.647922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:10.647971 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:10.648051 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:10.649341 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.649920 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.651023 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.651251 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.651334 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.654114 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:10.654118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:10.654121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:10.654151 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:10.655695 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:10.656341 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:10.656344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:10.656345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:10.656455 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:10.656791 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:10.656793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:10.656794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:10.656992 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:10.657116 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:10.657646 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 04:33:10.657755 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260620 04:33:10.658847 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.646145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:33287)
0620 04:33:10.646848 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:10.646893 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:10.647687 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:10.648211 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:10.648218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:10.648613 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:33:10.649177 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.649193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.650023 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.650025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.650916 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.650925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.651398 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.651401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.651546 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.652149 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:33:10.652165 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:33:10.653897 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:33:10.655794 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:10.655799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:10.655812 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:10.656210 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:10.656560 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:10.656562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:10.656564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:10.656710 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:10.657121 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:33:10.657127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:33:10.657384 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 04:33:10.657563 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 04:33:10.657902 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":124,"threads_started":1}
I20260620 04:33:10.660084 16247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41243
I20260620 04:33:10.660323 16247 heartbeater.cc:464] Registering TS with master...
I20260620 04:33:10.660830 16247 heartbeater.cc:511] Master 127.0.98.62:41243 requested a full tablet report, sending...
I20260620 04:33:10.661901 15788 ts_manager.cc:194] Registered new tserver with Master: 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603)
I20260620 04:33:10.662452 15788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33287
I20260620 04:33:10.667408 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:33:10.676586 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.669441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.0.98.62:41243)
0620 04:33:10.669564 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:10.669569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:10.669727 (+ 158us) server_negotiation.cc:407] Connection header received
0620 04:33:10.669871 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:10.669874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:10.669919 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:10.669984 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:10.670676 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.671292 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.672125 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.672402 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.672493 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.673780 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:10.673784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:10.673787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:10.673813 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:10.675047 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:10.675439 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:10.675442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:10.675444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:10.675522 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:10.675754 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:10.675758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:10.675759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:10.675982 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:10.676095 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:10.676329 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 04:33:10.676434 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:33:10.679500 15788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:33:10.682565 15788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:33:10.697091 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.691509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46603 (local address 127.0.0.1:48346)
0620 04:33:10.692052 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:10.692065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:10.692145 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:10.693159 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:10.693162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:10.693178 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:33:10.693269 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.693275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.694470 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.694473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.695563 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.695571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.696540 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.696543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.696657 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.696694 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 04:33:10.696801 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 04:33:10.696917 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":116,"threads_started":1}
I20260620 04:33:10.697124 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.692120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.0.98.3:46603)
0620 04:33:10.692686 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:10.692697 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:10.692727 (+ 30us) server_negotiation.cc:407] Connection header received
0620 04:33:10.692798 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:10.692803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:10.692995 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:10.693159 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:33:10.693374 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.694331 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.695687 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.696423 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.696527 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.696669 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:10.696857 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 04:33:10.696959 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":123,"threads_started":1}
I20260620 04:33:10.697319 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.692336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.0.98.1:43213)
0620 04:33:10.692573 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:10.692586 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:10.692984 (+ 398us) server_negotiation.cc:407] Connection header received
0620 04:33:10.693079 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:10.693084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:10.693267 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:10.693402 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:33:10.693776 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.694723 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.695930 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.696451 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.696543 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.696679 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:10.697094 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 04:33:10.697186 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260620 04:33:10.697399 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.691425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.0.98.2:37125)
0620 04:33:10.691831 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:10.691843 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:10.691867 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:33:10.691945 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:10.691951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:10.692172 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:10.692332 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:33:10.692840 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.693915 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.695588 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.696423 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.696527 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.696669 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:10.697168 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 04:33:10.697258 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":118,"threads_started":1}
I20260620 04:33:10.697697 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.691955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43213 (local address 127.0.0.1:46170)
0620 04:33:10.692745 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:10.692756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:10.692836 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:10.693551 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:10.693554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:10.693570 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:33:10.693662 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.693667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.694835 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.694839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.695824 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.695833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.696736 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.696739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.696867 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.696916 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 04:33:10.696976 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 04:33:10.697579 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":525,"thread_start_us":74,"threads_started":1}
I20260620 04:33:10.697888 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.691207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37125 (local address 127.0.0.1:38490)
0620 04:33:10.691566 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:10.691624 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:10.691744 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:10.692550 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:10.692556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:10.692577 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:33:10.692708 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.692716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.694031 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.694034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.695465 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.695473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.696745 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.696748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.696898 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.696921 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:33:10.696986 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 04:33:10.697094 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":256,"thread_start_us":93,"threads_started":1}
I20260620 04:33:10.700366 15913 tablet_service.cc:1467] Processing CreateTablet for tablet 6f758590a76b43319ed92c91bc912dbb (DEFAULT_TABLE table=test-table [id=e3913b7e05924b5fb5a66766d5365855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:33:10.700492 16181 tablet_service.cc:1467] Processing CreateTablet for tablet 6f758590a76b43319ed92c91bc912dbb (DEFAULT_TABLE table=test-table [id=e3913b7e05924b5fb5a66766d5365855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:33:10.700492 16047 tablet_service.cc:1467] Processing CreateTablet for tablet 6f758590a76b43319ed92c91bc912dbb (DEFAULT_TABLE table=test-table [id=e3913b7e05924b5fb5a66766d5365855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:33:10.701385 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f758590a76b43319ed92c91bc912dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:33:10.701504 16047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f758590a76b43319ed92c91bc912dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:33:10.701510 16181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f758590a76b43319ed92c91bc912dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:33:10.707758 16264 tablet_bootstrap.cc:492] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Bootstrap starting.
I20260620 04:33:10.708812 16265 tablet_bootstrap.cc:492] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: Bootstrap starting.
I20260620 04:33:10.709233 16266 tablet_bootstrap.cc:492] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: Bootstrap starting.
I20260620 04:33:10.710268 16264 tablet_bootstrap.cc:654] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Neither blocks nor log segments found. Creating new log.
I20260620 04:33:10.710896 16266 tablet_bootstrap.cc:654] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: Neither blocks nor log segments found. Creating new log.
I20260620 04:33:10.711158 16264 log.cc:824] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Log is configured to *not* fsync() on all Append() calls
I20260620 04:33:10.711328 16265 tablet_bootstrap.cc:654] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: Neither blocks nor log segments found. Creating new log.
I20260620 04:33:10.711975 16266 log.cc:824] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: Log is configured to *not* fsync() on all Append() calls
I20260620 04:33:10.712028 16265 log.cc:824] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:33:10.713697 16264 tablet_bootstrap.cc:492] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: No bootstrap required, opened a new log
I20260620 04:33:10.713791 16265 tablet_bootstrap.cc:492] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: No bootstrap required, opened a new log
I20260620 04:33:10.713792 16266 tablet_bootstrap.cc:492] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: No bootstrap required, opened a new log
I20260620 04:33:10.713893 16264 ts_tablet_manager.cc:1397] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:33:10.713966 16266 ts_tablet_manager.cc:1397] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 04:33:10.713966 16265 ts_tablet_manager.cc:1397] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:33:10.717280 16264 raft_consensus.cc:348] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:10.717525 16264 raft_consensus.cc:374] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:33:10.717576 16264 raft_consensus.cc:729] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c690c8b83242238ea732260f883fe4, State: Initialized, Role: FOLLOWER
I20260620 04:33:10.717707 16266 raft_consensus.cc:348] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:10.717756 16265 raft_consensus.cc:348] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:10.718025 16266 raft_consensus.cc:374] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:33:10.718025 16265 raft_consensus.cc:374] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:33:10.718079 16266 raft_consensus.cc:729] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8aedc4954a47b5b50150798daea3ba, State: Initialized, Role: FOLLOWER
I20260620 04:33:10.718079 16265 raft_consensus.cc:729] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf115ee8455e4fb286d8799f4c9d469f, State: Initialized, Role: FOLLOWER
I20260620 04:33:10.718079 16264 consensus_queue.cc:260] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:10.718528 16266 consensus_queue.cc:260] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:10.718528 16265 consensus_queue.cc:260] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:10.718806 15979 heartbeater.cc:503] Master 127.0.98.62:41243 was elected leader, sending a full tablet report...
I20260620 04:33:10.719115 16264 ts_tablet_manager.cc:1428] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:33:10.719537 16265 ts_tablet_manager.cc:1428] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:33:10.719767 16113 heartbeater.cc:503] Master 127.0.98.62:41243 was elected leader, sending a full tablet report...
I20260620 04:33:10.721156 16247 heartbeater.cc:503] Master 127.0.98.62:41243 was elected leader, sending a full tablet report...
I20260620 04:33:10.721437 16266 ts_tablet_manager.cc:1428] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:33:10.871856 16272 raft_consensus.cc:482] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:33:10.872072 16272 raft_consensus.cc:504] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:10.873297 16272 leader_election.cc:288] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125), 83c690c8b83242238ea732260f883fe4 (127.0.98.1:43213)
I20260620 04:33:10.877118 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.873524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37125 (local address 127.0.98.3:51435)
0620 04:33:10.873645 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:10.873661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:10.873759 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:10.874112 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:10.874115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:10.874133 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:33:10.874249 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.874255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.875092 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.875095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.875966 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.875976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.876633 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.876637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.876761 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.876781 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:33:10.876849 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 04:33:10.876922 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":34}
I20260620 04:33:10.877115 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.873629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51435 (local address 127.0.98.2:37125)
0620 04:33:10.873750 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:10.873754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:10.873772 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:33:10.873855 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:10.873859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:10.873913 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:10.874004 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:33:10.874364 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.874991 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.876087 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.876524 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.876601 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.876651 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:10.876890 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 04:33:10.876966 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:33:10.877166 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.873782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43213 (local address 127.0.98.3:53253)
0620 04:33:10.874121 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:10.874134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:10.874213 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:10.874418 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:10.874421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:10.874436 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:33:10.874546 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.874552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.875223 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.875226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.876094 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:10.876104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.876773 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.876776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:10.876879 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.876902 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:33:10.876967 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 04:33:10.877048 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":15,"thread_start_us":158,"threads_started":1}
I20260620 04:33:10.877175 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.873952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53253 (local address 127.0.98.1:43213)
0620 04:33:10.874091 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:10.874096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:10.874198 (+ 102us) server_negotiation.cc:407] Connection header received
0620 04:33:10.874282 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:10.874285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:10.874334 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:10.874396 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:33:10.874631 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.875118 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.876213 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.876651 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.876742 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.876802 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:10.876984 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 04:33:10.877076 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 04:33:10.877770 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f758590a76b43319ed92c91bc912dbb" candidate_uuid: "6d8aedc4954a47b5b50150798daea3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf115ee8455e4fb286d8799f4c9d469f" is_pre_election: true
I20260620 04:33:10.877785 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f758590a76b43319ed92c91bc912dbb" candidate_uuid: "6d8aedc4954a47b5b50150798daea3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c690c8b83242238ea732260f883fe4" is_pre_election: true
I20260620 04:33:10.878104 15933 raft_consensus.cc:2454] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8aedc4954a47b5b50150798daea3ba in term 0.
I20260620 04:33:10.878108 16067 raft_consensus.cc:2454] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8aedc4954a47b5b50150798daea3ba in term 0.
I20260620 04:33:10.878638 16137 leader_election.cc:302] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d8aedc4954a47b5b50150798daea3ba, 83c690c8b83242238ea732260f883fe4; no voters:
I20260620 04:33:10.878931 16272 raft_consensus.cc:2781] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:33:10.879010 16272 raft_consensus.cc:482] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:33:10.879061 16272 raft_consensus.cc:3037] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:33:10.880056 16272 raft_consensus.cc:504] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:10.880358 16272 leader_election.cc:288] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [CANDIDATE]: Term 1 election: Requested vote from peers cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125), 83c690c8b83242238ea732260f883fe4 (127.0.98.1:43213)
I20260620 04:33:10.880718 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f758590a76b43319ed92c91bc912dbb" candidate_uuid: "6d8aedc4954a47b5b50150798daea3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf115ee8455e4fb286d8799f4c9d469f"
I20260620 04:33:10.880748 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f758590a76b43319ed92c91bc912dbb" candidate_uuid: "6d8aedc4954a47b5b50150798daea3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c690c8b83242238ea732260f883fe4"
I20260620 04:33:10.880856 15933 raft_consensus.cc:3037] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:33:10.880856 16067 raft_consensus.cc:3037] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:33:10.882139 15933 raft_consensus.cc:2454] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8aedc4954a47b5b50150798daea3ba in term 1.
I20260620 04:33:10.882150 16067 raft_consensus.cc:2454] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8aedc4954a47b5b50150798daea3ba in term 1.
I20260620 04:33:10.882426 16137 leader_election.cc:302] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d8aedc4954a47b5b50150798daea3ba, 83c690c8b83242238ea732260f883fe4; no voters:
I20260620 04:33:10.882571 16272 raft_consensus.cc:2781] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:33:10.882793 16272 raft_consensus.cc:686] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 1 LEADER]: Becoming Leader. State: Replica: 6d8aedc4954a47b5b50150798daea3ba, State: Running, Role: LEADER
I20260620 04:33:10.883109 16272 consensus_queue.cc:237] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:10.886170 15787 catalog_manager.cc:5495] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6d8aedc4954a47b5b50150798daea3ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } health_report { overall_health: HEALTHY } } }
W20260620 04:33:10.894660 16248 tablet.cc:2369] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:33:10.913417 16114 tablet.cc:2369] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:33:10.932534 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.929140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.0.98.3:46603)
0620 04:33:10.929274 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:10.929279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:10.929291 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:33:10.929367 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:10.929369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:10.929423 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:10.929520 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:33:10.929780 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.930274 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.931115 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:10.931327 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:10.931413 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:10.931709 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:33:10.932131 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:33:10.932240 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:10.932309 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 04:33:10.932365 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 04:33:10.937100 15980 tablet.cc:2369] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:33:10.938581 15933 raft_consensus.cc:1261] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Refusing update from remote peer 6d8aedc4954a47b5b50150798daea3ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:33:10.938581 16067 raft_consensus.cc:1261] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f [term 1 FOLLOWER]: Refusing update from remote peer 6d8aedc4954a47b5b50150798daea3ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:33:10.939291 16272 consensus_queue.cc:1035] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:33:10.939481 16274 consensus_queue.cc:1035] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:33:10.948303 16279 mvcc.cc:205] Tried to move back new op lower bound from 7298785242876329984 to 7298785242659889152. Current Snapshot: MvccSnapshot[applied={T|T < 7298785242876329984}]
I20260620 04:33:10.956576 15788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:33:10.956766 15788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:33:10.959219 15788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Sending DeleteTablet for 3 replicas of tablet 6f758590a76b43319ed92c91bc912dbb
I20260620 04:33:10.960099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 15850
I20260620 04:33:10.960155 16181 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f758590a76b43319ed92c91bc912dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:33:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48346
I20260620 04:33:10.960165 16047 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f758590a76b43319ed92c91bc912dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:33:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38490
I20260620 04:33:10.960842 16287 tablet_replica.cc:330] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: stopping tablet replica
I20260620 04:33:10.961201 16287 raft_consensus.cc:2229] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:33:10.961522 16287 raft_consensus.cc:2258] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:33:10.961558 16286 tablet_replica.cc:330] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: stopping tablet replica
I20260620 04:33:10.961956 16286 raft_consensus.cc:2229] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:33:10.962463 16286 raft_consensus.cc:2258] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:33:10.963119 16287 ts_tablet_manager.cc:1905] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:33:10.963742 16286 ts_tablet_manager.cc:1905] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:33:10.965863 16287 ts_tablet_manager.cc:1918] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:33:10.965976 16287 log.cc:1196] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/6f758590a76b43319ed92c91bc912dbb
I20260620 04:33:10.966269 16287 ts_tablet_manager.cc:1939] T 6f758590a76b43319ed92c91bc912dbb P cf115ee8455e4fb286d8799f4c9d469f: Deleting consensus metadata
I20260620 04:33:10.966578 16286 ts_tablet_manager.cc:1918] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:33:10.966670 16286 log.cc:1196] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/6f758590a76b43319ed92c91bc912dbb
I20260620 04:33:10.967038 16286 ts_tablet_manager.cc:1939] T 6f758590a76b43319ed92c91bc912dbb P 6d8aedc4954a47b5b50150798daea3ba: Deleting consensus metadata
W20260620 04:33:10.967480 15775 connection.cc:535] client connection to 127.0.98.1:43213 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:33:10.967577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 15984
W20260620 04:33:10.967644 15775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:33:10.971024 15773 catalog_manager.cc:4841] TS 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603): tablet 6f758590a76b43319ed92c91bc912dbb (table test-table [id=e3913b7e05924b5fb5a66766d5365855]) successfully deleted
I20260620 04:33:10.971307 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.971040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43213 (local address 127.0.0.1:46174)
0620 04:33:10.971155 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:10.971225 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 04:33:10.971494 15775 catalog_manager.cc:4584] TS 83c690c8b83242238ea732260f883fe4 (127.0.98.1:43213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f758590a76b43319ed92c91bc912dbb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43213: connect: Connection refused (error 111)
I20260620 04:33:10.973747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 16118
I20260620 04:33:10.975505 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:10.974034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37125 (local address 127.0.0.1:38502)
0620 04:33:10.975369 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:10.975420 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1257}
I20260620 04:33:10.979246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 15756
I20260620 04:33:30.984968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41243
--webserver_interface=127.0.98.62
--webserver_port=45615
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:33:31.095865 16289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:31.096150 16289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:31.096199 16289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:31.100209 16289 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 04:33:31.100262 16289 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 04:33:31.100277 16289 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 04:33:31.100291 16289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:33:31.100307 16289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:31.100324 16289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:31.100349 16289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:33:31.100378 16289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:33:31.105798 16289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:31.106885 16289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:31.107928 16289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:31.120918 16296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:31.120981 16289 server_base.cc:1034] running on GCE node
W20260620 04:33:31.120918 16295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:31.120918 16298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:31.121759 16289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:31.123080 16289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:31.124253 16289 hybrid_clock.cc:648] HybridClock initialized: now 1781930011124239 us; error 30 us; skew 500 ppm
Jun 20 04:33:31 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:31.127140 16289 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:33:31.128100 16289 webserver.cc:460] Webserver started at http://127.0.98.62:45615/ using document root <none> and password file <none>
I20260620 04:33:31.128710 16289 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:31.128791 16289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:31.132921 16289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:33:31.135264 16305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:31.136559 16289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:33:31.136667 16289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "a658e635bae74110a620ebba15b66a15"
format_stamp: "Formatted at 2026-06-20 04:33:09 on dist-test-slave-rzqm"
server_key: "531fc9d51146ce5fb2c861f8273d956f"
server_key_iv: "dd94649c9fec9e099a7bab8b0276ef63"
server_key_version: "encryptionkey@0"
I20260620 04:33:31.137315 16289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:31.153584 16289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:31.156620 16289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:31.156806 16289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:31.163554 16289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41243
I20260620 04:33:31.163808 16357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41243 every 8 connection(s)
I20260620 04:33:31.164620 16289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:33:31.169085 16358 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:33:31.170854 16358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Bootstrap starting.
I20260620 04:33:31.171610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 16289
I20260620 04:33:31.172084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43213
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41243
--webserver_port=45649
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:33:31.188836 16358 log.cc:824] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Log is configured to *not* fsync() on all Append() calls
I20260620 04:33:31.197754 16358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:33:31.198127 16358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Bootstrap complete.
I20260620 04:33:31.201963 16358 raft_consensus.cc:348] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:31.202342 16358 raft_consensus.cc:729] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a658e635bae74110a620ebba15b66a15, State: Initialized, Role: FOLLOWER
I20260620 04:33:31.202805 16358 consensus_queue.cc:260] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:31.202917 16358 raft_consensus.cc:388] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:33:31.202982 16358 raft_consensus.cc:482] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:33:31.203073 16358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 1 FOLLOWER]: Advancing to term 2
I20260620 04:33:31.204929 16358 raft_consensus.cc:504] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:31.205287 16358 leader_election.cc:302] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a658e635bae74110a620ebba15b66a15; no voters:
I20260620 04:33:31.205581 16358 leader_election.cc:288] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 04:33:31.205672 16362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 04:33:31.205869 16362 raft_consensus.cc:686] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 2 LEADER]: Becoming Leader. State: Replica: a658e635bae74110a620ebba15b66a15, State: Running, Role: LEADER
I20260620 04:33:31.206247 16362 consensus_queue.cc:237] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:31.206804 16358 sys_catalog.cc:564] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:33:31.207180 16362 sys_catalog.cc:455] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a658e635bae74110a620ebba15b66a15. Latest consensus state: current_term: 2 leader_uuid: "a658e635bae74110a620ebba15b66a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } } }
I20260620 04:33:31.207288 16362 sys_catalog.cc:458] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: This master's current role is: LEADER
I20260620 04:33:31.207635 16366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:33:31.207664 16363 sys_catalog.cc:455] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a658e635bae74110a620ebba15b66a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } } }
I20260620 04:33:31.207887 16363 sys_catalog.cc:458] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: This master's current role is: LEADER
I20260620 04:33:31.212435 16366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:33:31.213099 16366 catalog_manager.cc:1259] Loaded cluster ID: 7249911cc34c407d936a2751d661332b
I20260620 04:33:31.213160 16366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:33:31.215559 16366 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:33:31.216317 16366 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Loaded TSK: 0
I20260620 04:33:31.230628 16366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Generated new TSK 1
I20260620 04:33:31.230916 16366 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:33:31.285741 16360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:31.286022 16360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:31.286082 16360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:31.290078 16360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:31.290136 16360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:31.290211 16360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:33:31.295444 16360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45649
--tserver_master_addrs=127.0.98.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:31.296451 16360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:31.297490 16360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:31.307643 16385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:31.307643 16386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:31.307741 16388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:31.308092 16360 server_base.cc:1034] running on GCE node
I20260620 04:33:31.308492 16360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:31.309196 16360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:31.310391 16360 hybrid_clock.cc:648] HybridClock initialized: now 1781930011310363 us; error 44 us; skew 500 ppm
Jun 20 04:33:31 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:31.313234 16360 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:33:31.314117 16360 webserver.cc:460] Webserver started at http://127.0.98.1:45649/ using document root <none> and password file <none>
I20260620 04:33:31.314642 16360 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:31.314721 16360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:31.318756 16360 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 04:33:31.320820 16395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:31.322064 16360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:33:31.322165 16360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "83c690c8b83242238ea732260f883fe4"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "8e3ba8dd39fd697b6619902a75527611"
server_key_iv: "c13ca6d386310f23a1a81f8327800e3f"
server_key_version: "encryptionkey@0"
I20260620 04:33:31.322695 16360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:31.351167 16360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:31.354311 16360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:31.354482 16360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:31.355065 16360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:33:31.356148 16402 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:33:31.360215 16360 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:33:31.360282 16360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 04:33:31.360342 16360 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:33:31.362052 16360 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:33:31.362094 16360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 04:33:31.362421 16402 tablet_bootstrap.cc:492] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Bootstrap starting.
I20260620 04:33:31.371615 16360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43213
I20260620 04:33:31.371706 16509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43213 every 8 connection(s)
I20260620 04:33:31.372545 16360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:33:31.378845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 16360
I20260620 04:33:31.379312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37125
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41243
--webserver_port=44269
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:33:31.380306 16402 log.cc:824] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Log is configured to *not* fsync() on all Append() calls
Jun 20 04:33:31 dist-test-slave-rzqm krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:33:31.388968 16402 tablet_bootstrap.cc:492] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:33:31.389390 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.374512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51629 (local address 127.0.98.62:41243)
0620 04:33:31.374857 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.374874 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.375260 (+ 386us) server_negotiation.cc:407] Connection header received
0620 04:33:31.376318 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.376328 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.376620 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.376930 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:31.378092 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.379126 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.380358 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.380994 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.381185 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.383747 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:31.383752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:31.383764 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:31.383793 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:31.386295 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.386791 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.386795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.386799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.386917 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.387229 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.387232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.387233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.387569 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:31.387696 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:31.388219 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 04:33:31.388477 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
I20260620 04:33:31.389420 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.374617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:51629)
0620 04:33:31.375089 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:31.375139 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:31.375920 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:31.377032 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:31.377040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:31.377467 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:33:31.377915 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.377930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.379270 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.379273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.380233 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.380243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.381158 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.381163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.381365 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.382005 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:33:31.382023 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:33:31.383551 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:33:31.386413 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:31.386418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:31.386433 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:31.386683 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:31.387005 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:31.387008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:31.387010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:31.387137 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:31.387675 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:33:31.387681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:33:31.387930 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 04:33:31.388087 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 04:33:31.388436 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":123,"threads_started":1}
I20260620 04:33:31.389437 16402 tablet_bootstrap.cc:492] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Bootstrap complete.
I20260620 04:33:31.391182 16402 ts_tablet_manager.cc:1397] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.015s
I20260620 04:33:31.391767 16510 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41243
I20260620 04:33:31.391983 16510 heartbeater.cc:464] Registering TS with master...
I20260620 04:33:31.392442 16510 heartbeater.cc:511] Master 127.0.98.62:41243 requested a full tablet report, sending...
I20260620 04:33:31.394192 16402 raft_consensus.cc:348] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:31.394721 16402 raft_consensus.cc:729] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c690c8b83242238ea732260f883fe4, State: Initialized, Role: FOLLOWER
I20260620 04:33:31.394734 16322 ts_manager.cc:194] Registered new tserver with Master: 83c690c8b83242238ea732260f883fe4 (127.0.98.1:43213)
I20260620 04:33:31.395195 16402 consensus_queue.cc:260] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } } peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } }
I20260620 04:33:31.396175 16402 ts_tablet_manager.cc:1428] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 04:33:31.398092 16322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51629
I20260620 04:33:31.400580 16510 heartbeater.cc:503] Master 127.0.98.62:41243 was elected leader, sending a full tablet report...
Jun 20 04:33:31 dist-test-slave-rzqm krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 04:33:31.408654 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.397705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.0.98.1:43213)
0620 04:33:31.398893 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.398903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.398922 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:33:31.398975 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.398979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.399157 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.399286 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:31.399982 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.400977 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.402234 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.402559 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.402637 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.404973 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:31.404981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:31.404991 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:31.405039 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:31.406703 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.407226 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.407232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.407234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.407317 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.407575 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.407578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.407580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.407836 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:31.407976 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:31.408332 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 04:33:31.408493 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":79,"threads_started":1}
I20260620 04:33:31.408654 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.397635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43213 (local address 127.0.0.1:47592)
0620 04:33:31.398498 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:31.398546 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:31.398671 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:31.399306 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:31.399311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:31.399727 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:33:31.399859 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.399866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.401124 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.401128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.402113 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.402124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.402646 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.402663 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.402778 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.403347 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:33:31.403361 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:33:31.404838 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:33:31.406822 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:31.406832 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:31.406838 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:31.407115 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:31.407405 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:31.407408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:31.407410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:31.407494 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:31.407990 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:33:31.407997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:33:31.408113 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 04:33:31.408238 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 04:33:31.408451 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":117,"threads_started":1}
I20260620 04:33:31.410094 16444 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f758590a76b43319ed92c91bc912dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:33:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47592
I20260620 04:33:31.410918 16519 tablet_replica.cc:330] stopping tablet replica
I20260620 04:33:31.411190 16519 raft_consensus.cc:2229] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:33:31.411437 16519 raft_consensus.cc:2258] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:33:31.412330 16519 ts_tablet_manager.cc:1905] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:33:31.415740 16519 ts_tablet_manager.cc:1918] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:33:31.415823 16519 log.cc:1196] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal/wals/6f758590a76b43319ed92c91bc912dbb
I20260620 04:33:31.416030 16519 ts_tablet_manager.cc:1939] T 6f758590a76b43319ed92c91bc912dbb P 83c690c8b83242238ea732260f883fe4: Deleting consensus metadata
I20260620 04:33:31.417816 16309 catalog_manager.cc:4841] TS 83c690c8b83242238ea732260f883fe4 (127.0.98.1:43213): tablet 6f758590a76b43319ed92c91bc912dbb (table test-table [id=e3913b7e05924b5fb5a66766d5365855]) successfully deleted
W20260620 04:33:31.497727 16514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:31.497999 16514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:31.498078 16514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:31.502100 16514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:31.502168 16514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:31.502250 16514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:33:31.507601 16514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44269
--tserver_master_addrs=127.0.98.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:31.508741 16514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:31.509809 16514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:33:31.519346 16514 server_base.cc:1034] running on GCE node
W20260620 04:33:31.519383 16525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:31.519508 16527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:31.519374 16524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:31.519924 16514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:31.520538 16514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:31.521780 16514 hybrid_clock.cc:648] HybridClock initialized: now 1781930011521760 us; error 28 us; skew 500 ppm
Jun 20 04:33:31 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:31.528425 16514 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:33:31.529362 16514 webserver.cc:460] Webserver started at http://127.0.98.2:44269/ using document root <none> and password file <none>
I20260620 04:33:31.529907 16514 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:31.529985 16514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:31.533980 16514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:33:31.536046 16534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:31.537410 16514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:33:31.537514 16514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "cf115ee8455e4fb286d8799f4c9d469f"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "425fbf5ec548f30430539f883196c4ea"
server_key_iv: "cd61e4d3a859b556bbd79c613553d5bc"
server_key_version: "encryptionkey@0"
I20260620 04:33:31.538045 16514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:31.557492 16514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:31.560436 16514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:31.560611 16514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:31.561230 16514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:33:31.562052 16514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:33:31.562098 16514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:31.562148 16514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:33:31.562176 16514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:31.571053 16514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37125
I20260620 04:33:31.571135 16647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37125 every 8 connection(s)
I20260620 04:33:31.572077 16514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:33:31.577338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 16514
I20260620 04:33:31.577811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46603
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41243
--webserver_port=43037
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:33:31 dist-test-slave-rzqm krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:33:31.585758 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.573962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44899 (local address 127.0.98.62:41243)
0620 04:33:31.574106 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.574110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.575215 (+ 1105us) server_negotiation.cc:407] Connection header received
0620 04:33:31.576069 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.576072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.576123 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.576196 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:31.577598 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.578348 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.579518 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.579758 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.579851 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.582422 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:31.582425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:31.582427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:31.582455 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:31.583741 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.584217 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.584220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.584222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.584302 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.584612 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.584615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.584617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.584815 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:31.584899 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:31.585448 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 04:33:31.585588 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:33:31.586613 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.574406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:44899)
0620 04:33:31.575065 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:31.575106 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:31.575878 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:31.576422 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:31.576430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:31.576826 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:33:31.577428 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.577444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.578468 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.578471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.579386 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.579396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.579904 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.579908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.580071 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.580633 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:33:31.580649 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:33:31.582259 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:33:31.583850 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:31.583856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:31.583872 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:31.584124 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:31.584382 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:31.584385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:31.584387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:31.584528 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:31.584913 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:33:31.584920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:33:31.585218 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 04:33:31.585392 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 04:33:31.585747 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":102,"threads_started":1}
I20260620 04:33:31.587937 16648 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41243
I20260620 04:33:31.588205 16648 heartbeater.cc:464] Registering TS with master...
I20260620 04:33:31.588698 16648 heartbeater.cc:511] Master 127.0.98.62:41243 requested a full tablet report, sending...
I20260620 04:33:31.589797 16322 ts_manager.cc:194] Registered new tserver with Master: cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125)
I20260620 04:33:31.590566 16322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44899
W20260620 04:33:31.687216 16651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:31.687469 16651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:31.687510 16651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:31.691447 16651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:31.691505 16651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:31.691562 16651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:33:31.696743 16651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43037
--tserver_master_addrs=127.0.98.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:31.697775 16651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:31.698714 16651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:31.708182 16659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:31.708158 16656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:31.708350 16651 server_base.cc:1034] running on GCE node
W20260620 04:33:31.708036 16657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:31.708865 16651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:31.709566 16651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:31.710757 16651 hybrid_clock.cc:648] HybridClock initialized: now 1781930011710742 us; error 29 us; skew 500 ppm
Jun 20 04:33:31 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:31.713419 16651 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:33:31.714293 16651 webserver.cc:460] Webserver started at http://127.0.98.3:43037/ using document root <none> and password file <none>
I20260620 04:33:31.714859 16651 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:31.714936 16651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:31.718904 16651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:33:31.720890 16666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:31.722086 16651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:33:31.722190 16651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "6d8aedc4954a47b5b50150798daea3ba"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "0eee212138622a97c926a6efe647dcce"
server_key_iv: "b790f90dc228b2d5333ae640d197716d"
server_key_version: "encryptionkey@0"
I20260620 04:33:31.722707 16651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:31.737190 16651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:31.739935 16651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:31.740103 16651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:31.740720 16651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:33:31.741657 16651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:33:31.741703 16651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:31.741753 16651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:33:31.741780 16651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:31.750137 16651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46603
I20260620 04:33:31.750181 16779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46603 every 8 connection(s)
I20260620 04:33:31.751101 16651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:33:31.753871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 16651
Jun 20 04:33:31 dist-test-slave-rzqm krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:33:31.761924 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.754517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.0.98.62:41243)
0620 04:33:31.754720 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.754724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.754910 (+ 186us) server_negotiation.cc:407] Connection header received
0620 04:33:31.755031 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.755034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.755081 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.755165 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:31.755879 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.756421 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.757265 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.757481 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.757565 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.758896 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:31.758900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:31.758902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:31.758929 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:31.760206 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.760582 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.760586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.760588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.760670 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.760924 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.760927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.760929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.761206 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:31.761419 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:31.761572 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 04:33:31.761752 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":54,"threads_started":1}
I20260620 04:33:31.764055 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.752905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51477 (local address 127.0.98.62:41243)
0620 04:33:31.753069 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.753073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.754120 (+ 1047us) server_negotiation.cc:407] Connection header received
0620 04:33:31.755154 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.755157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.755203 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.755277 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:31.756390 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.756858 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.757979 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.758183 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.758273 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.760645 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:31.760649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:31.760651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:31.760674 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:31.762186 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.762668 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.762670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.762672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.762747 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.763076 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.763079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.763080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.763237 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:31.763308 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:31.763854 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 04:33:31.763947 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:33:31.764837 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.753319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:51477)
0620 04:33:31.753961 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:31.754017 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:31.754963 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:31.755445 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:31.755452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:31.755795 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:33:31.756258 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.756273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.756966 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.756969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.757871 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.757880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.758314 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.758318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.758477 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.758989 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:33:31.759008 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:33:31.760495 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:33:31.762285 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:31.762290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:31.762306 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:31.762585 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:31.762822 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:31.762824 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:31.762826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:31.762991 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:31.763328 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:33:31.763334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:33:31.763624 (+ 290us) client_negotiation.cc:769] Sending connection context
0620 04:33:31.763794 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 04:33:31.764033 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":111,"threads_started":1}
I20260620 04:33:31.765933 16780 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41243
I20260620 04:33:31.766142 16780 heartbeater.cc:464] Registering TS with master...
I20260620 04:33:31.766588 16780 heartbeater.cc:511] Master 127.0.98.62:41243 requested a full tablet report, sending...
I20260620 04:33:31.767565 16322 ts_manager.cc:194] Registered new tserver with Master: 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603)
I20260620 04:33:31.768071 16322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51477
I20260620 04:33:31.775359 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:33:31.779369 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.775971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.0.98.62:41243)
0620 04:33:31.776116 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.776121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.776273 (+ 152us) server_negotiation.cc:407] Connection header received
0620 04:33:31.776432 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.776434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.776531 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.776619 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:33:31.776874 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.777476 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.778305 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.778509 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.778592 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.778869 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:33:31.779003 (+ 134us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 04:33:31.779247 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44982: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 04:33:31.779435 16512 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44982: token expired
W20260620 04:33:31.779711 16253 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:41243)
I20260620 04:33:31.786429 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.780036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.0.98.62:41243)
0620 04:33:31.780196 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.780200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.780211 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:33:31.780244 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.780247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.780285 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.780343 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:31.780931 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.781464 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.782203 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.782419 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.782498 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.783681 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:31.783685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:31.783687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:31.783713 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:31.784923 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.785331 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.785335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.785337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.785410 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:31.785643 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:31.785647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:31.785649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:31.785816 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:31.785932 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:31.786136 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 04:33:31.786285 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 04:33:31.788723 16322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:33:31.791877 16322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:33:31.799506 16444 tablet_service.cc:1467] Processing CreateTablet for tablet d8aecac257ac4b2aa0eb824d4f2c4aa2 (DEFAULT_TABLE table=test-table [id=b20e6fe8c0b14cd7b4e31268a9b04925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:33:31.800175 16444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8aecac257ac4b2aa0eb824d4f2c4aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:33:31.802654 16402 tablet_bootstrap.cc:492] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Bootstrap starting.
I20260620 04:33:31.803534 16402 tablet_bootstrap.cc:654] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Neither blocks nor log segments found. Creating new log.
I20260620 04:33:31.804134 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.798315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46603 (local address 127.0.0.1:34054)
0620 04:33:31.798416 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:31.798431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:31.798509 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:31.799716 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:31.799719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:31.799736 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:33:31.799837 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.799843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.801165 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.801168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.802494 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.802503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.803715 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.803718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.803814 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.803859 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 04:33:31.803912 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 04:33:31.803997 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":20}
I20260620 04:33:31.804276 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.798533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37125 (local address 127.0.0.1:57628)
0620 04:33:31.799201 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:31.799215 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:31.799315 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:31.799958 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:31.799960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:31.799977 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:33:31.800058 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.800064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.801305 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.801308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.802491 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.802503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.803900 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.803903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.803981 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.803999 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:33:31.804066 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 04:33:31.804123 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":53,"thread_start_us":144,"threads_started":1}
I20260620 04:33:31.804656 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.798725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.0.98.2:37125)
0620 04:33:31.799028 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.799039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.799305 (+ 266us) server_negotiation.cc:407] Connection header received
0620 04:33:31.799422 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.799428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.799625 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.799782 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:33:31.800176 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.801209 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.802627 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.803566 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.803684 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.803841 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:31.804121 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 04:33:31.804531 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":107,"threads_started":1}
I20260620 04:33:31.804837 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.798801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.0.98.3:46603)
0620 04:33:31.799257 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.799269 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.799292 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:33:31.799369 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.799375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.799566 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.799843 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:33:31.800023 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.801064 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.802627 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.803595 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.804430 (+ 835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.804565 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:31.804615 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 04:33:31.804703 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":130,"threads_started":1}
I20260620 04:33:31.804988 16402 tablet_bootstrap.cc:492] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: No bootstrap required, opened a new log
I20260620 04:33:31.805117 16402 ts_tablet_manager.cc:1397] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260620 04:33:31.805557 16402 raft_consensus.cc:348] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:31.805689 16402 raft_consensus.cc:374] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:33:31.805732 16402 raft_consensus.cc:729] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c690c8b83242238ea732260f883fe4, State: Initialized, Role: FOLLOWER
I20260620 04:33:31.805838 16402 consensus_queue.cc:260] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:31.806062 16402 ts_tablet_manager.cc:1428] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 04:33:31.807884 16582 tablet_service.cc:1467] Processing CreateTablet for tablet d8aecac257ac4b2aa0eb824d4f2c4aa2 (DEFAULT_TABLE table=test-table [id=b20e6fe8c0b14cd7b4e31268a9b04925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:33:31.808027 16714 tablet_service.cc:1467] Processing CreateTablet for tablet d8aecac257ac4b2aa0eb824d4f2c4aa2 (DEFAULT_TABLE table=test-table [id=b20e6fe8c0b14cd7b4e31268a9b04925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:33:31.808790 16582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8aecac257ac4b2aa0eb824d4f2c4aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:33:31.808815 16714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8aecac257ac4b2aa0eb824d4f2c4aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:33:31.815632 16793 tablet_bootstrap.cc:492] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: Bootstrap starting.
I20260620 04:33:31.816195 16794 tablet_bootstrap.cc:492] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: Bootstrap starting.
I20260620 04:33:31.818285 16794 tablet_bootstrap.cc:654] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: Neither blocks nor log segments found. Creating new log.
I20260620 04:33:31.818384 16793 tablet_bootstrap.cc:654] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: Neither blocks nor log segments found. Creating new log.
I20260620 04:33:31.819289 16793 log.cc:824] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:33:31.819383 16794 log.cc:824] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: Log is configured to *not* fsync() on all Append() calls
I20260620 04:33:31.821468 16793 tablet_bootstrap.cc:492] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: No bootstrap required, opened a new log
I20260620 04:33:31.821468 16794 tablet_bootstrap.cc:492] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: No bootstrap required, opened a new log
I20260620 04:33:31.821666 16793 ts_tablet_manager.cc:1397] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:33:31.821668 16794 ts_tablet_manager.cc:1397] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:33:31.825590 16794 raft_consensus.cc:348] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:31.825590 16793 raft_consensus.cc:348] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:31.825906 16793 raft_consensus.cc:374] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:33:31.825906 16794 raft_consensus.cc:374] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:33:31.825959 16793 raft_consensus.cc:729] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf115ee8455e4fb286d8799f4c9d469f, State: Initialized, Role: FOLLOWER
I20260620 04:33:31.825959 16794 raft_consensus.cc:729] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8aedc4954a47b5b50150798daea3ba, State: Initialized, Role: FOLLOWER
I20260620 04:33:31.826450 16793 consensus_queue.cc:260] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:31.826450 16794 consensus_queue.cc:260] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:31.827153 16648 heartbeater.cc:503] Master 127.0.98.62:41243 was elected leader, sending a full tablet report...
I20260620 04:33:31.827570 16794 ts_tablet_manager.cc:1428] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:33:31.827570 16793 ts_tablet_manager.cc:1428] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:33:31.828243 16780 heartbeater.cc:503] Master 127.0.98.62:41243 was elected leader, sending a full tablet report...
W20260620 04:33:31.873771 16511 tablet.cc:2369] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:33:31.970541 16798 raft_consensus.cc:482] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:33:31.970767 16798 raft_consensus.cc:504] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:31.971949 16798 leader_election.cc:288] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125), 83c690c8b83242238ea732260f883fe4 (127.0.98.1:43213)
I20260620 04:33:31.976253 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.972200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37125 (local address 127.0.98.3:44505)
0620 04:33:31.972314 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:31.972327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:31.972429 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:31.972694 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:31.972698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:31.972722 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:33:31.972864 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.972872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.973936 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.973939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.975111 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.975121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.975914 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.975917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.975999 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.976018 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:33:31.976073 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 04:33:31.976130 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":11}
I20260620 04:33:31.976307 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.972318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44505 (local address 127.0.98.2:37125)
0620 04:33:31.972452 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.972456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.972473 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:33:31.972526 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.972529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.972576 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.972658 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:33:31.973033 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.973825 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.975244 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.975799 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.975905 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.975970 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:31.976109 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 04:33:31.976185 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:33:31.976656 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.972491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43213 (local address 127.0.98.3:43569)
0620 04:33:31.972881 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:31.972894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:31.972979 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:31.973292 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:31.973294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:31.973308 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:33:31.973397 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.973401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.974284 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.974288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.975357 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:31.975365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.976296 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.976299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:31.976387 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.976408 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:33:31.976471 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:33:31.976533 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":121,"threads_started":1}
I20260620 04:33:31.976679 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:31.972627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43569 (local address 127.0.98.1:43213)
0620 04:33:31.972875 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:31.972878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:31.972973 (+ 95us) server_negotiation.cc:407] Connection header received
0620 04:33:31.973084 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:31.973087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:31.973159 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:31.973267 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:33:31.973503 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.974146 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.975473 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:31.976140 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:31.976244 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:31.976408 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:31.976488 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 04:33:31.976561 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":103,"threads_started":1}
I20260620 04:33:31.976980 16602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aecac257ac4b2aa0eb824d4f2c4aa2" candidate_uuid: "6d8aedc4954a47b5b50150798daea3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf115ee8455e4fb286d8799f4c9d469f" is_pre_election: true
I20260620 04:33:31.977295 16464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aecac257ac4b2aa0eb824d4f2c4aa2" candidate_uuid: "6d8aedc4954a47b5b50150798daea3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c690c8b83242238ea732260f883fe4" is_pre_election: true
I20260620 04:33:31.977366 16602 raft_consensus.cc:2454] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8aedc4954a47b5b50150798daea3ba in term 0.
I20260620 04:33:31.977536 16464 raft_consensus.cc:2454] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d8aedc4954a47b5b50150798daea3ba in term 0.
I20260620 04:33:31.977860 16670 leader_election.cc:302] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d8aedc4954a47b5b50150798daea3ba, cf115ee8455e4fb286d8799f4c9d469f; no voters:
I20260620 04:33:31.978120 16798 raft_consensus.cc:2781] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:33:31.978199 16798 raft_consensus.cc:482] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:33:31.978228 16798 raft_consensus.cc:3037] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:33:31.979218 16798 raft_consensus.cc:504] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:31.979523 16798 leader_election.cc:288] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [CANDIDATE]: Term 1 election: Requested vote from peers cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125), 83c690c8b83242238ea732260f883fe4 (127.0.98.1:43213)
I20260620 04:33:31.979899 16602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aecac257ac4b2aa0eb824d4f2c4aa2" candidate_uuid: "6d8aedc4954a47b5b50150798daea3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf115ee8455e4fb286d8799f4c9d469f"
I20260620 04:33:31.979930 16464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aecac257ac4b2aa0eb824d4f2c4aa2" candidate_uuid: "6d8aedc4954a47b5b50150798daea3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c690c8b83242238ea732260f883fe4"
I20260620 04:33:31.980003 16602 raft_consensus.cc:3037] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:33:31.980062 16464 raft_consensus.cc:3037] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:33:31.980990 16602 raft_consensus.cc:2454] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8aedc4954a47b5b50150798daea3ba in term 1.
I20260620 04:33:31.981042 16464 raft_consensus.cc:2454] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d8aedc4954a47b5b50150798daea3ba in term 1.
I20260620 04:33:31.981336 16670 leader_election.cc:302] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d8aedc4954a47b5b50150798daea3ba, cf115ee8455e4fb286d8799f4c9d469f; no voters:
I20260620 04:33:31.981483 16798 raft_consensus.cc:2781] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:33:31.981673 16798 raft_consensus.cc:686] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 1 LEADER]: Becoming Leader. State: Replica: 6d8aedc4954a47b5b50150798daea3ba, State: Running, Role: LEADER
I20260620 04:33:31.982010 16798 consensus_queue.cc:237] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:31.985039 16322 catalog_manager.cc:5495] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6d8aedc4954a47b5b50150798daea3ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } health_report { overall_health: UNKNOWN } } }
W20260620 04:33:32.002113 16781 tablet.cc:2369] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:33:32.038743 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:32.035496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.0.98.3:46603)
0620 04:33:32.035632 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:32.035636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:32.035651 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:33:32.035777 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:32.035779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:32.035839 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:32.035924 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:33:32.036104 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:32.036595 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:32.037463 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:32.037668 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:32.037748 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:32.037982 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:33:32.038387 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:33:32.038468 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:32.038550 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 04:33:32.038603 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:33:32.044564 16602 raft_consensus.cc:1261] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f [term 1 FOLLOWER]: Refusing update from remote peer 6d8aedc4954a47b5b50150798daea3ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:33:32.044560 16464 raft_consensus.cc:1261] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Refusing update from remote peer 6d8aedc4954a47b5b50150798daea3ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:33:32.045228 16798 consensus_queue.cc:1035] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:33:32.045462 16801 consensus_queue.cc:1035] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:33:32.051410 16804 mvcc.cc:205] Tried to move back new op lower bound from 7298785329327558656 to 7298785329080778752. Current Snapshot: MvccSnapshot[applied={T|T < 7298785329327558656}]
I20260620 04:33:32.060698 16322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:33:32.060889 16322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:33:32.063279 16322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Sending DeleteTablet for 3 replicas of tablet d8aecac257ac4b2aa0eb824d4f2c4aa2
I20260620 04:33:32.064003 16714 tablet_service.cc:1514] Processing DeleteTablet for tablet d8aecac257ac4b2aa0eb824d4f2c4aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:33:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34054
I20260620 04:33:32.064105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 16360
I20260620 04:33:32.064174 16582 tablet_service.cc:1514] Processing DeleteTablet for tablet d8aecac257ac4b2aa0eb824d4f2c4aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:33:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57628
I20260620 04:33:32.064931 16813 tablet_replica.cc:330] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: stopping tablet replica
I20260620 04:33:32.065315 16813 raft_consensus.cc:2229] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:33:32.065652 16813 raft_consensus.cc:2258] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:33:32.065872 16812 tablet_replica.cc:330] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: stopping tablet replica
I20260620 04:33:32.066210 16812 raft_consensus.cc:2229] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:33:32.066612 16812 raft_consensus.cc:2258] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:33:32.067348 16813 ts_tablet_manager.cc:1905] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:33:32.068317 16812 ts_tablet_manager.cc:1905] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:33:32.070125 16813 ts_tablet_manager.cc:1918] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:33:32.070232 16813 log.cc:1196] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/d8aecac257ac4b2aa0eb824d4f2c4aa2
I20260620 04:33:32.070578 16813 ts_tablet_manager.cc:1939] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P cf115ee8455e4fb286d8799f4c9d469f: Deleting consensus metadata
W20260620 04:33:32.070932 16309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43213 (error 108)
I20260620 04:33:32.071451 16812 ts_tablet_manager.cc:1918] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:33:32.071609 16812 log.cc:1196] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/d8aecac257ac4b2aa0eb824d4f2c4aa2
I20260620 04:33:32.071985 16812 ts_tablet_manager.cc:1939] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 6d8aedc4954a47b5b50150798daea3ba: Deleting consensus metadata
I20260620 04:33:32.071978 16309 catalog_manager.cc:4841] TS cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125): tablet d8aecac257ac4b2aa0eb824d4f2c4aa2 (table test-table [id=b20e6fe8c0b14cd7b4e31268a9b04925]) successfully deleted
I20260620 04:33:32.072054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 16514
I20260620 04:33:32.073510 16307 catalog_manager.cc:4841] TS 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603): tablet d8aecac257ac4b2aa0eb824d4f2c4aa2 (table test-table [id=b20e6fe8c0b14cd7b4e31268a9b04925]) successfully deleted
I20260620 04:33:32.077484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 16651
I20260620 04:33:32.078990 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:32.078641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43213 (local address 127.0.0.1:47594)
0620 04:33:32.078825 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:32.078902 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 04:33:32.079164 16309 catalog_manager.cc:4584] TS 83c690c8b83242238ea732260f883fe4 (127.0.98.1:43213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8aecac257ac4b2aa0eb824d4f2c4aa2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43213: connect: Connection refused (error 111)
I20260620 04:33:32.082697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 16289
I20260620 04:33:52.088486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41243
--webserver_interface=127.0.98.62
--webserver_port=45615
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:33:52.197686 16815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:52.197956 16815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:52.198015 16815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:52.202018 16815 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 04:33:52.202072 16815 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 04:33:52.202093 16815 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 04:33:52.202116 16815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:33:52.202142 16815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:52.202174 16815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:52.202205 16815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:33:52.202234 16815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:33:52.207551 16815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:52.208619 16815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:52.209686 16815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:52.217690 16820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:52.217746 16823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:52.217736 16821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:52.217871 16815 server_base.cc:1034] running on GCE node
I20260620 04:33:52.218497 16815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:52.219555 16815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:52.220727 16815 hybrid_clock.cc:648] HybridClock initialized: now 1781930032220701 us; error 37 us; skew 500 ppm
Jun 20 04:33:52 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:52.223420 16815 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:33:52.224347 16815 webserver.cc:460] Webserver started at http://127.0.98.62:45615/ using document root <none> and password file <none>
I20260620 04:33:52.224880 16815 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:52.224941 16815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:52.228648 16815 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:33:52.230886 16830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:52.232043 16815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:33:52.232143 16815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "a658e635bae74110a620ebba15b66a15"
format_stamp: "Formatted at 2026-06-20 04:33:09 on dist-test-slave-rzqm"
server_key: "531fc9d51146ce5fb2c861f8273d956f"
server_key_iv: "dd94649c9fec9e099a7bab8b0276ef63"
server_key_version: "encryptionkey@0"
I20260620 04:33:52.232676 16815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:52.259871 16815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:52.263106 16815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:52.263285 16815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:52.270259 16815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41243
I20260620 04:33:52.270576 16882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41243 every 8 connection(s)
I20260620 04:33:52.271348 16815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:33:52.274748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 16815
I20260620 04:33:52.275167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43213
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41243
--webserver_port=45649
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:33:52.275556 16883 sys_catalog.cc:263] Verifying existing consensus state
I20260620 04:33:52.277122 16883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Bootstrap starting.
I20260620 04:33:52.293519 16883 log.cc:824] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Log is configured to *not* fsync() on all Append() calls
I20260620 04:33:52.304113 16883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:33:52.304454 16883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Bootstrap complete.
I20260620 04:33:52.308188 16883 raft_consensus.cc:348] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:52.308580 16883 raft_consensus.cc:729] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a658e635bae74110a620ebba15b66a15, State: Initialized, Role: FOLLOWER
I20260620 04:33:52.309098 16883 consensus_queue.cc:260] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:52.309226 16883 raft_consensus.cc:388] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:33:52.309293 16883 raft_consensus.cc:482] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:33:52.309387 16883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 2 FOLLOWER]: Advancing to term 3
I20260620 04:33:52.311195 16883 raft_consensus.cc:504] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:52.311501 16883 leader_election.cc:302] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a658e635bae74110a620ebba15b66a15; no voters:
I20260620 04:33:52.311781 16883 leader_election.cc:288] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 04:33:52.311877 16887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 04:33:52.312065 16887 raft_consensus.cc:686] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [term 3 LEADER]: Becoming Leader. State: Replica: a658e635bae74110a620ebba15b66a15, State: Running, Role: LEADER
I20260620 04:33:52.312425 16887 consensus_queue.cc:237] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } }
I20260620 04:33:52.313042 16883 sys_catalog.cc:564] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:33:52.313730 16888 sys_catalog.cc:455] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a658e635bae74110a620ebba15b66a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } } }
I20260620 04:33:52.313849 16888 sys_catalog.cc:458] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: This master's current role is: LEADER
I20260620 04:33:52.314215 16889 sys_catalog.cc:455] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a658e635bae74110a620ebba15b66a15. Latest consensus state: current_term: 3 leader_uuid: "a658e635bae74110a620ebba15b66a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658e635bae74110a620ebba15b66a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41243 } } }
I20260620 04:33:52.314345 16889 sys_catalog.cc:458] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15 [sys.catalog]: This master's current role is: LEADER
I20260620 04:33:52.314611 16895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:33:52.319819 16895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:33:52.320490 16895 catalog_manager.cc:1259] Loaded cluster ID: 7249911cc34c407d936a2751d661332b
I20260620 04:33:52.320547 16895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:33:52.322916 16895 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:33:52.323601 16895 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Loaded TSK: 0
I20260620 04:33:52.323680 16895 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a658e635bae74110a620ebba15b66a15: Loaded TSK: 1
F20260620 04:33:52.324388 16895 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa9a5e649ad google::LogMessage::Fail() at ??:0
@ 0x7fa9a5e6788a google::LogMessage::SendToLog() at ??:0
@ 0x7fa9a5e644de google::LogMessage::Flush() at ??:0
@ 0x7fa9a5e65a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa9a8b2818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa9a8b20f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa9b07c97bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa9b07a399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa9b07a4178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fa9b07e5df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fa9b080482b std::function<>::operator()() at ??:0
@ 0x7fa9b07a3b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa9b07a4f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa9b07a1579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa9b07e5640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa9a80b63d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa9a80b6c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa9a80b83f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa9a80a7512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa9a96eb6db start_thread at ??:0
@ 0x7fa9a496261f clone at ??:0
W20260620 04:33:52.413028 16885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:52.413378 16885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:52.413439 16885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:52.419405 16885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:52.419468 16885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:52.419554 16885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:33:52.425740 16885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45649
--tserver_master_addrs=127.0.98.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:52.426831 16885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:52.427850 16885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:52.437306 16930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:52.437342 16931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:52.437341 16885 server_base.cc:1034] running on GCE node
W20260620 04:33:52.437548 16933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:52.438066 16885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:52.438692 16885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:52.439864 16885 hybrid_clock.cc:648] HybridClock initialized: now 1781930032439837 us; error 37 us; skew 500 ppm
Jun 20 04:33:52 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:52.447464 16885 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:33:52.448421 16885 webserver.cc:460] Webserver started at http://127.0.98.1:45649/ using document root <none> and password file <none>
I20260620 04:33:52.448984 16885 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:52.449098 16885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:52.453343 16885 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 04:33:52.457113 16940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:52.458504 16885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:33:52.458623 16885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "83c690c8b83242238ea732260f883fe4"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "8e3ba8dd39fd697b6619902a75527611"
server_key_iv: "c13ca6d386310f23a1a81f8327800e3f"
server_key_version: "encryptionkey@0"
I20260620 04:33:52.459177 16885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:52.483283 16885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:52.486423 16885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:52.486598 16885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:52.487288 16885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:33:52.488519 16947 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 04:33:52.493952 16885 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 04:33:52.494030 16885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 04:33:52.494100 16885 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 04:33:52.495805 16885 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 04:33:52.495853 16885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 04:33:52.496174 16947 tablet_bootstrap.cc:492] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Bootstrap starting.
I20260620 04:33:52.505451 16885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43213
I20260620 04:33:52.505682 17054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43213 every 8 connection(s)
I20260620 04:33:52.506459 16885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:33:52.511732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 16885
I20260620 04:33:52.512151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37125
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41243
--webserver_port=44269
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:33:52.517572 16947 log.cc:824] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Log is configured to *not* fsync() on all Append() calls
I20260620 04:33:52.527189 16947 tablet_bootstrap.cc:492] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 04:33:52.527604 16947 tablet_bootstrap.cc:492] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Bootstrap complete.
I20260620 04:33:52.528913 16947 ts_tablet_manager.cc:1397] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.007s
I20260620 04:33:52.532238 16947 raft_consensus.cc:348] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:52.532792 16947 raft_consensus.cc:729] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c690c8b83242238ea732260f883fe4, State: Initialized, Role: FOLLOWER
I20260620 04:33:52.533367 16947 consensus_queue.cc:260] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:52.534502 16947 ts_tablet_manager.cc:1428] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260620 04:33:52.543670 17057 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:33:52.508376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:47197)
0620 04:33:52.508873 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:52.508939 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:52.509786 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:52.542926 (+ 33140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
W20260620 04:33:52.544153 17055 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41243 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41243: Client connection negotiation failed: client connection to 127.0.98.62:41243: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:33:52.544817 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:52.544599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:53047)
0620 04:33:52.544710 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:52.544766 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 04:33:52.545444 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:52.545247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:52103)
0620 04:33:52.545348 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:52.545389 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 04:33:52.545584 17055 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 04:33:52.630923 17058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:52.631254 17058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:52.631322 17058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:52.635455 17058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:52.635524 17058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:52.635607 17058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:33:52.640915 17058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44269
--tserver_master_addrs=127.0.98.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:52.641973 17058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:52.642961 17058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:52.652149 17068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:52.652149 17066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:52.652155 17065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:52.652367 17058 server_base.cc:1034] running on GCE node
I20260620 04:33:52.652830 17058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:52.653506 17058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:52.654685 17058 hybrid_clock.cc:648] HybridClock initialized: now 1781930032654659 us; error 35 us; skew 500 ppm
Jun 20 04:33:52 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:52.657446 17058 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:33:52.658330 17058 webserver.cc:460] Webserver started at http://127.0.98.2:44269/ using document root <none> and password file <none>
I20260620 04:33:52.658890 17058 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:52.658953 17058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:52.663020 17058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:33:52.665180 17075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:52.666410 17058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:33:52.666508 17058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "cf115ee8455e4fb286d8799f4c9d469f"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "425fbf5ec548f30430539f883196c4ea"
server_key_iv: "cd61e4d3a859b556bbd79c613553d5bc"
server_key_version: "encryptionkey@0"
I20260620 04:33:52.667061 17058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:52.680778 17058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:52.683815 17058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:52.683982 17058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:52.684546 17058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:33:52.685376 17058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:33:52.685420 17058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:52.685458 17058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:33:52.685472 17058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:52.693809 17058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37125
I20260620 04:33:52.693830 17188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37125 every 8 connection(s)
I20260620 04:33:52.694749 17058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:33:52.698316 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:52.696755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:41269)
0620 04:33:52.697495 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:52.697605 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":123,"threads_started":1}
W20260620 04:33:52.698741 17189 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41243 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41243: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
I20260620 04:33:52.699323 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:52.699115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:36711)
0620 04:33:52.699218 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:52.699270 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 04:33:52.699913 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:52.699671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:55229)
0620 04:33:52.699811 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:52.699861 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 04:33:52.700034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 17058
W20260620 04:33:52.700142 17189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:33:52.700449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46603
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41243
--webserver_port=43037
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:33:52.807839 17192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:33:52.808087 17192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:33:52.808130 17192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:33:52.812162 17192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:33:52.812245 17192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:33:52.812305 17192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:33:52.817559 17192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43037
--tserver_master_addrs=127.0.98.62:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:33:52.818574 17192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:33:52.819568 17192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:33:52.828754 17200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:52.828841 17198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:33:52.828841 17197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:33:52.829265 17192 server_base.cc:1034] running on GCE node
I20260620 04:33:52.829656 17192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:33:52.830272 17192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:33:52.831418 17192 hybrid_clock.cc:648] HybridClock initialized: now 1781930032831409 us; error 56 us; skew 500 ppm
Jun 20 04:33:52 dist-test-slave-rzqm krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:33:52.834162 17192 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:33:52.835014 17192 webserver.cc:460] Webserver started at http://127.0.98.3:43037/ using document root <none> and password file <none>
I20260620 04:33:52.835543 17192 fs_manager.cc:359] Metadata directory not provided
I20260620 04:33:52.835608 17192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:33:52.839515 17192 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:33:52.841620 17207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:52.842809 17192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:33:52.842913 17192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "6d8aedc4954a47b5b50150798daea3ba"
format_stamp: "Formatted at 2026-06-20 04:33:10 on dist-test-slave-rzqm"
server_key: "0eee212138622a97c926a6efe647dcce"
server_key_iv: "b790f90dc228b2d5333ae640d197716d"
server_key_version: "encryptionkey@0"
I20260620 04:33:52.843448 17192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:33:52.866935 17192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:33:52.870025 17192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:33:52.870225 17192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:33:52.870831 17192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:33:52.871656 17192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:33:52.871703 17192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:52.871753 17192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:33:52.871780 17192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:33:52.880043 17192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46603
I20260620 04:33:52.880079 17320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46603 every 8 connection(s)
I20260620 04:33:52.880971 17192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:33:52.884419 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:52.883019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:39305)
0620 04:33:52.883650 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:52.883756 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":107,"threads_started":1}
W20260620 04:33:52.884866 17321 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41243 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41243: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
I20260620 04:33:52.885401 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:52.885209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:53405)
0620 04:33:52.885308 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:52.885351 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 04:33:52.885914 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:52.885759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:60383)
0620 04:33:52.885835 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:52.885871 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 04:33:52.886058 17321 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 04:33:52.886562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 17192
I20260620 04:33:53.546665 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:53.546148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:45469)
0620 04:33:53.546448 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:53.546541 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260620 04:33:53.701087 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:53.700654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:52915)
0620 04:33:53.700885 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:53.700937 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260620 04:33:53.886983 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:53.886579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:43807)
0620 04:33:53.886815 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:53.886883 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
I20260620 04:33:54.110322 17328 raft_consensus.cc:482] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:33:54.110543 17328 raft_consensus.cc:504] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:54.111630 17328 leader_election.cc:288] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603), cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125)
Jun 20 04:33:54 dist-test-slave-rzqm krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 04:33:54 dist-test-slave-rzqm krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 04:33:54.128163 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:54.112142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57369 (local address 127.0.98.2:37125)
0620 04:33:54.112452 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:54.112477 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:54.112524 (+ 47us) server_negotiation.cc:407] Connection header received
0620 04:33:54.113443 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:54.113451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:54.113915 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:54.114250 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:54.115968 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:54.117548 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:54.119278 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:54.120032 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:54.120186 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:54.123235 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:54.123242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:54.123262 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:54.123297 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:54.125211 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:54.125833 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:54.125840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:54.125846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:54.125990 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:54.126467 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:54.126471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:54.126472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:54.126957 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:54.127072 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:54.127693 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 04:33:54.127979 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":53,"threads_started":1}
I20260620 04:33:54.128157 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:54.112015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45313 (local address 127.0.98.3:46603)
0620 04:33:54.112452 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 04:33:54.112478 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 04:33:54.112527 (+ 49us) server_negotiation.cc:407] Connection header received
0620 04:33:54.113443 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:33:54.113451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:33:54.113914 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:33:54.114250 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:33:54.115968 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:54.117579 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:54.119278 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:33:54.120007 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:33:54.120186 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:54.123570 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:33:54.123576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:33:54.123592 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:33:54.123628 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:33:54.125697 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:54.126223 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:54.126229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:54.126236 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:54.126365 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:33:54.126684 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:33:54.126686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:33:54.126688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:33:54.127135 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:33:54.127248 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 04:33:54.127697 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 04:33:54.127955 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":97,"threads_started":1}
I20260620 04:33:54.128152 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:54.111887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46603 (local address 127.0.98.1:45313)
0620 04:33:54.112256 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:54.112272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:54.112377 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:54.114376 (+ 1999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:54.114383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:54.115000 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:33:54.115775 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:54.115797 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:54.117713 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:54.117717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:54.119156 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:54.119169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:54.120160 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:54.120164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:54.120417 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:54.121151 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:33:54.121163 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:33:54.123303 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:33:54.125821 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:54.125824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:54.125826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:54.126065 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:54.126474 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:54.126478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:54.126480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:54.126597 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:54.127296 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:33:54.127299 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:33:54.127384 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 04:33:54.127560 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 04:33:54.127975 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":231,"spinlock_wait_cycles":14848,"thread_start_us":74,"threads_started":1}
I20260620 04:33:54.128782 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:54.111887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37125 (local address 127.0.98.1:57369)
0620 04:33:54.112340 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:54.112351 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:33:54.112427 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:33:54.114376 (+ 1949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:33:54.114383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:33:54.115000 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:33:54.115775 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:54.115797 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:54.117713 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:54.117717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:54.119157 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:33:54.119169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:33:54.120160 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:33:54.120164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:33:54.120446 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:33:54.121114 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:33:54.121136 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:33:54.122983 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:33:54.125331 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:54.125339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:54.125355 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:54.125692 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:54.126172 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:33:54.126176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:33:54.126178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:33:54.126361 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:33:54.127101 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:33:54.127108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:33:54.127366 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 04:33:54.127589 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 04:33:54.128054 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":551,"spinlock_wait_cycles":11648,"thread_start_us":66,"threads_started":1}
W20260620 04:33:54.130833 16942 leader_election.cc:341] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603): Not found: Tablet not found: d8aecac257ac4b2aa0eb824d4f2c4aa2
W20260620 04:33:54.130980 16944 leader_election.cc:341] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125): Not found: Tablet not found: d8aecac257ac4b2aa0eb824d4f2c4aa2
I20260620 04:33:54.131052 16944 leader_election.cc:302] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c690c8b83242238ea732260f883fe4; no voters: 6d8aedc4954a47b5b50150798daea3ba, cf115ee8455e4fb286d8799f4c9d469f
I20260620 04:33:54.131302 17328 raft_consensus.cc:2726] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:33:54.547693 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:54.547407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:59971)
0620 04:33:54.547546 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:54.547610 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 04:33:54.702193 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:54.701789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:56615)
0620 04:33:54.702045 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:54.702104 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260620 04:33:54.888108 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:54.887703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:51473)
0620 04:33:54.887959 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:54.888019 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260620 04:33:55.548843 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:55.548394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:44597)
0620 04:33:55.548695 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:55.548749 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":115,"threads_started":1}
I20260620 04:33:55.677259 17336 raft_consensus.cc:482] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:33:55.677389 17336 raft_consensus.cc:504] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:55.677788 17336 leader_election.cc:288] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603), cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125)
W20260620 04:33:55.678562 16942 leader_election.cc:341] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603): Not found: Tablet not found: d8aecac257ac4b2aa0eb824d4f2c4aa2
W20260620 04:33:55.678696 16944 leader_election.cc:341] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125): Not found: Tablet not found: d8aecac257ac4b2aa0eb824d4f2c4aa2
I20260620 04:33:55.678797 16944 leader_election.cc:302] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c690c8b83242238ea732260f883fe4; no voters: 6d8aedc4954a47b5b50150798daea3ba, cf115ee8455e4fb286d8799f4c9d469f
I20260620 04:33:55.678942 17336 raft_consensus.cc:2726] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:33:55.703354 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:55.702918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:60915)
0620 04:33:55.703191 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:55.703250 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
I20260620 04:33:55.889283 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:55.888849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:34761)
0620 04:33:55.889113 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:55.889173 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
I20260620 04:33:56.550055 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:56.549620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:35569)
0620 04:33:56.549867 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:56.549930 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260620 04:33:56.704505 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:56.704097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:59281)
0620 04:33:56.704358 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:56.704415 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260620 04:33:56.890362 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:56.889960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:47025)
0620 04:33:56.890216 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:56.890272 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260620 04:33:57.551657 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:57.551062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:40945)
0620 04:33:57.551458 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:57.551536 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":78,"threads_started":1}
I20260620 04:33:57.686714 17343 raft_consensus.cc:482] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:33:57.686848 17343 raft_consensus.cc:504] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:57.687225 17343 leader_election.cc:288] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603), cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125)
W20260620 04:33:57.687963 16944 leader_election.cc:341] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125): Not found: Tablet not found: d8aecac257ac4b2aa0eb824d4f2c4aa2
W20260620 04:33:57.688141 16942 leader_election.cc:341] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603): Not found: Tablet not found: d8aecac257ac4b2aa0eb824d4f2c4aa2
I20260620 04:33:57.688232 16942 leader_election.cc:302] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c690c8b83242238ea732260f883fe4; no voters: 6d8aedc4954a47b5b50150798daea3ba, cf115ee8455e4fb286d8799f4c9d469f
I20260620 04:33:57.688401 17343 raft_consensus.cc:2726] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:33:57.705740 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:57.705293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:59959)
0620 04:33:57.705592 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:57.705649 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":102,"threads_started":1}
I20260620 04:33:57.891639 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:57.891147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:56683)
0620 04:33:57.891455 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:57.891534 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
I20260620 04:33:58.552938 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:58.552512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:53201)
0620 04:33:58.552781 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:58.552841 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260620 04:33:58.706795 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:58.706429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:42231)
0620 04:33:58.706659 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:58.706709 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260620 04:33:58.892731 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:58.892366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:37069)
0620 04:33:58.892583 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:58.892638 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260620 04:33:59.398128 17349 raft_consensus.cc:482] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:33:59.398303 17349 raft_consensus.cc:504] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:33:59.398738 17349 leader_election.cc:288] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603), cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125)
W20260620 04:33:59.399482 16942 leader_election.cc:341] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603): Not found: Tablet not found: d8aecac257ac4b2aa0eb824d4f2c4aa2
W20260620 04:33:59.399580 16944 leader_election.cc:341] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125): Not found: Tablet not found: d8aecac257ac4b2aa0eb824d4f2c4aa2
I20260620 04:33:59.399690 16944 leader_election.cc:302] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c690c8b83242238ea732260f883fe4; no voters: 6d8aedc4954a47b5b50150798daea3ba, cf115ee8455e4fb286d8799f4c9d469f
I20260620 04:33:59.399842 17349 raft_consensus.cc:2726] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:33:59.554142 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:59.553720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:54265)
0620 04:33:59.553986 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:59.554048 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
I20260620 04:33:59.708002 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:59.707512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:58291)
0620 04:33:59.707847 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:59.707908 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":76,"threads_started":1}
I20260620 04:33:59.894019 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:33:59.893523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:35707)
0620 04:33:59.893859 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 04:33:59.893921 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":74,"threads_started":1}
I20260620 04:34:00.555452 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:00.555015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:34453)
0620 04:34:00.555288 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:00.555355 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260620 04:34:00.709326 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:00.708786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:39801)
0620 04:34:00.709135 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:00.709206 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260620 04:34:00.895296 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:00.894840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:46265)
0620 04:34:00.895104 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:00.895196 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260620 04:34:01.556723 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:01.556285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:34333)
0620 04:34:01.556562 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:01.556627 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260620 04:34:01.710785 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:01.710217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:43039)
0620 04:34:01.710577 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:01.710669 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":93,"threads_started":1}
I20260620 04:34:01.783990 17358 raft_consensus.cc:482] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:34:01.784163 17358 raft_consensus.cc:504] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8aedc4954a47b5b50150798daea3ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46603 } } peers { permanent_uuid: "cf115ee8455e4fb286d8799f4c9d469f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37125 } } peers { permanent_uuid: "83c690c8b83242238ea732260f883fe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43213 } }
I20260620 04:34:01.784675 17358 leader_election.cc:288] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603), cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125)
W20260620 04:34:01.785478 16942 leader_election.cc:341] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d8aedc4954a47b5b50150798daea3ba (127.0.98.3:46603): Not found: Tablet not found: d8aecac257ac4b2aa0eb824d4f2c4aa2
W20260620 04:34:01.785602 16944 leader_election.cc:341] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf115ee8455e4fb286d8799f4c9d469f (127.0.98.2:37125): Not found: Tablet not found: d8aecac257ac4b2aa0eb824d4f2c4aa2
I20260620 04:34:01.785720 16944 leader_election.cc:302] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c690c8b83242238ea732260f883fe4; no voters: 6d8aedc4954a47b5b50150798daea3ba, cf115ee8455e4fb286d8799f4c9d469f
I20260620 04:34:01.785883 17358 raft_consensus.cc:2726] T d8aecac257ac4b2aa0eb824d4f2c4aa2 P 83c690c8b83242238ea732260f883fe4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 04:34:01.896625 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:01.896172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:54505)
0620 04:34:01.896461 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:01.896528 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260620 04:34:02.557951 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:02.557528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:41843)
0620 04:34:02.557797 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:02.557856 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260620 04:34:02.712193 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:02.711731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:46719)
0620 04:34:02.712017 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:02.712083 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260620 04:34:02.897925 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:02.897498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:35201)
0620 04:34:02.897757 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:02.897824 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260620 04:34:03.559125 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:03.558711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:52643)
0620 04:34:03.558965 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:03.559030 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260620 04:34:03.713480 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:03.712982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:44343)
0620 04:34:03.713320 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:03.713384 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260620 04:34:03.899143 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:03.898699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:53935)
0620 04:34:03.898985 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:03.899047 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
I20260620 04:34:04.560333 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:04.559853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:41353)
0620 04:34:04.560169 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:04.560233 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260620 04:34:04.714836 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:04.714358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:44629)
0620 04:34:04.714639 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:04.714738 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260620 04:34:04.900462 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:04.899997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:60263)
0620 04:34:04.900305 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:04.900369 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":88,"threads_started":1}
I20260620 04:34:05.561570 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:05.561153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:53865)
0620 04:34:05.561414 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:05.561475 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
I20260620 04:34:05.716027 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:05.715589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:59405)
0620 04:34:05.715862 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:05.715920 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
I20260620 04:34:05.901877 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:05.901384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:42553)
0620 04:34:05.901681 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:05.901734 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
I20260620 04:34:06.562788 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:06.562355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:58295)
0620 04:34:06.562626 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:06.562689 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260620 04:34:06.717308 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:06.716825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:43505)
0620 04:34:06.717148 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:06.717213 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":75,"threads_started":1}
I20260620 04:34:06.903074 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:06.902619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:51953)
0620 04:34:06.902915 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:06.902979 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260620 04:34:07.564167 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:07.563736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.1:32967)
0620 04:34:07.564002 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:07.564062 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260620 04:34:07.718570 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:07.718128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.2:48579)
0620 04:34:07.718407 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:07.718468 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260620 04:34:07.896615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 16885
I20260620 04:34:07.901603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 17058
I20260620 04:34:07.905238 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:07.904761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41243 (local address 127.0.98.3:54213)
0620 04:34:07.905052 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:07.905128 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":65,"threads_started":1}
I20260620 04:34:07.908185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 17192
2026-06-20T04:34:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59258 ms)
[----------] 30 tests from SecurityITest (240245 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:07 dist-test-slave-rzqm krb5kdc[17380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:07 dist-test-slave-rzqm krb5kdc[17380](info): set up 2 sockets
Jun 20 04:34:07 dist-test-slave-rzqm krb5kdc[17380](info): commencing operation
krb5kdc: starting...
W20260620 04:34:08.864629 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:08 dist-test-slave-rzqm krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:08.983851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41375
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:09.092651 17393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:09.092916 17393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:09.092954 17393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:09.096948 17393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:09.097031 17393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:09.097050 17393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:09.097065 17393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:09.097090 17393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:09.102423 17393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:09.103461 17393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:09.104480 17393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:09.112874 17398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:09.112860 17399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:09.113085 17393 server_base.cc:1034] running on GCE node
W20260620 04:34:09.112977 17401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:09.113675 17393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:09.114785 17393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:09.115952 17393 hybrid_clock.cc:648] HybridClock initialized: now 1781930049115923 us; error 40 us; skew 500 ppm
Jun 20 04:34:09 dist-test-slave-rzqm krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:09.122380 17393 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:09.123306 17393 webserver.cc:460] Webserver started at http://127.0.98.62:33265/ using document root <none> and password file <none>
I20260620 04:34:09.123864 17393 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:09.123927 17393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:09.124101 17393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:09.126273 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "ae5f4d2463704fc5853f44fd9aec0f4c"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "732a1692d505d9e71ac7d3fe3b960b65"
server_key_iv: "7c5bc5c25a2ff1e574a6d2d698b4a0fd"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.126828 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae5f4d2463704fc5853f44fd9aec0f4c"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "732a1692d505d9e71ac7d3fe3b960b65"
server_key_iv: "7c5bc5c25a2ff1e574a6d2d698b4a0fd"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.130731 17393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:34:09.133313 17408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:09.134545 17393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:09.134654 17393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "ae5f4d2463704fc5853f44fd9aec0f4c"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "732a1692d505d9e71ac7d3fe3b960b65"
server_key_iv: "7c5bc5c25a2ff1e574a6d2d698b4a0fd"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.134753 17393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:09.153342 17393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:09.156273 17393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:09.156426 17393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:09.163097 17393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41375
I20260620 04:34:09.163116 17460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41375 every 8 connection(s)
I20260620 04:34:09.164091 17393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:09.166806 17461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:09.169952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 17393
I20260620 04:34:09.170077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:09.170387 392 external_mini_cluster.cc:1428] Setting key 59003cb8ff2ff3cd30edf9d411bc214f
I20260620 04:34:09.172773 17461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c: Bootstrap starting.
I20260620 04:34:09.175292 17461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:09.176123 17461 log.cc:824] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:09.178289 17461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c: No bootstrap required, opened a new log
Jun 20 04:34:09 dist-test-slave-rzqm krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:09.181896 17461 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5f4d2463704fc5853f44fd9aec0f4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } }
I20260620 04:34:09.182112 17461 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:09.182193 17461 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5f4d2463704fc5853f44fd9aec0f4c, State: Initialized, Role: FOLLOWER
I20260620 04:34:09.182667 17461 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5f4d2463704fc5853f44fd9aec0f4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } }
I20260620 04:34:09.182802 17461 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:09.182860 17461 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:09.182974 17461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:09.183882 17461 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5f4d2463704fc5853f44fd9aec0f4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } }
I20260620 04:34:09.184191 17461 leader_election.cc:302] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae5f4d2463704fc5853f44fd9aec0f4c; no voters:
I20260620 04:34:09.184338 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.171776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36366 (local address 127.0.98.62:41375)
0620 04:34:09.172188 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:09.172205 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:09.172241 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:34:09.172892 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:09.172920 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:09.173320 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:09.173626 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:09.174518 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.175587 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.176693 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.177275 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.177391 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.179957 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:09.179962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:09.179976 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:09.180002 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:09.181565 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:09.181959 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:09.181964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:09.181969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:09.182080 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:09.182406 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:09.182409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:09.182410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:09.182789 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:09.183001 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:09.183277 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 04:34:09.183558 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":109,"threads_started":1}
I20260620 04:34:09.184502 17461 leader_election.cc:288] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:09.184834 17467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:09.185051 17467 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [term 1 LEADER]: Becoming Leader. State: Replica: ae5f4d2463704fc5853f44fd9aec0f4c, State: Running, Role: LEADER
I20260620 04:34:09.185421 17467 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5f4d2463704fc5853f44fd9aec0f4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } }
I20260620 04:34:09.185688 17461 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:09.186908 17466 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae5f4d2463704fc5853f44fd9aec0f4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5f4d2463704fc5853f44fd9aec0f4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } } }
I20260620 04:34:09.187124 17466 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:09.187414 17466 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5f4d2463704fc5853f44fd9aec0f4c. Latest consensus state: current_term: 1 leader_uuid: "ae5f4d2463704fc5853f44fd9aec0f4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5f4d2463704fc5853f44fd9aec0f4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41375 } } }
I20260620 04:34:09.187501 17466 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c [sys.catalog]: This master's current role is: LEADER
W20260620 04:34:09.190068 17481 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:09.190151 17481 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:09.190230 17478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:09.190908 17478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:09.195873 17478 catalog_manager.cc:1347] Generated new cluster ID: 00e928617fab4271bd9335b338c0c973
I20260620 04:34:09.195945 17478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:09.225991 17478 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:09.226997 17478 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:09.234958 17478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c: Generated new TSK 0
I20260620 04:34:09.235656 17478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:34:09.263518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:09.372670 17487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:09.372939 17487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:09.373023 17487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:09.376921 17487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:09.376979 17487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:09.377075 17487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:34:09.382159 17487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:09.383149 17487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:09.384133 17487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:09.394306 17495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:09.394325 17493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:09.394342 17492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:09.394554 17487 server_base.cc:1034] running on GCE node
I20260620 04:34:09.394995 17487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:09.395581 17487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:09.396755 17487 hybrid_clock.cc:648] HybridClock initialized: now 1781930049396742 us; error 26 us; skew 500 ppm
Jun 20 04:34:09 dist-test-slave-rzqm krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:09.399475 17487 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:34:09.400316 17487 webserver.cc:460] Webserver started at http://127.0.98.1:43267/ using document root <none> and password file <none>
I20260620 04:34:09.400866 17487 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:09.400934 17487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:09.401149 17487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:09.403141 17487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "46166574f4674f516ad7c1f2e1280d56"
server_key_iv: "96caf3a271556d66a2a698146c45e48f"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.403672 17487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "46166574f4674f516ad7c1f2e1280d56"
server_key_iv: "96caf3a271556d66a2a698146c45e48f"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.407387 17487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:34:09.409772 17502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:09.410966 17487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:09.411072 17487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "46166574f4674f516ad7c1f2e1280d56"
server_key_iv: "96caf3a271556d66a2a698146c45e48f"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.411165 17487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:09.422981 17487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:09.426015 17487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:09.426185 17487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:09.426738 17487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:09.427522 17487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:09.427578 17487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:09.427631 17487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:09.427657 17487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:09.436213 17487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36585
I20260620 04:34:09.436251 17615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36585 every 8 connection(s)
I20260620 04:34:09.437107 17487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:34:09.439709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 17487
I20260620 04:34:09.439836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:34:09.440150 392 external_mini_cluster.cc:1428] Setting key 6c3c4f5ede4d657b40fdebd8cb02277c
Jun 20 04:34:09 dist-test-slave-rzqm krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:34:09.450156 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.438848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45417 (local address 127.0.98.62:41375)
0620 04:34:09.438963 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:09.438966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:09.439936 (+ 970us) server_negotiation.cc:407] Connection header received
0620 04:34:09.440919 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:09.440922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:09.440975 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:09.441067 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:09.442293 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.442803 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.443917 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.444140 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.444232 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.446736 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:09.446739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:09.446741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:09.446764 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:09.448050 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:09.448604 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:09.448606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:09.448608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:09.448687 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:09.449030 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:09.449033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:09.449035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:09.449249 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:09.449329 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:09.449873 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 04:34:09.450027 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:34:09.450940 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.439258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41375 (local address 127.0.98.1:45417)
0620 04:34:09.439774 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:09.439831 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:09.440730 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:09.441245 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:09.441252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:09.441690 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:09.442154 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.442170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.442917 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.442920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.443804 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.443813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.444278 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.444283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.444450 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.445023 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:09.445045 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:09.446556 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:09.448168 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:09.448175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:09.448190 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:09.448516 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:09.448759 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:09.448761 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:09.448763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:09.448924 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:09.449347 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:09.449353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:09.449595 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 04:34:09.449752 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 04:34:09.450043 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260620 04:34:09.452245 17616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41375
I20260620 04:34:09.452524 17616 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:09.453171 17616 heartbeater.cc:511] Master 127.0.98.62:41375 requested a full tablet report, sending...
I20260620 04:34:09.454725 17425 ts_manager.cc:194] Registered new tserver with Master: cdb1737a14ab4f76bf619d4d7bbecfa0 (127.0.98.1:36585)
I20260620 04:34:09.455844 17425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:34:09.465415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:09.573535 17621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:09.573807 17621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:09.573849 17621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:09.577780 17621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:09.577836 17621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:09.577893 17621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:34:09.583035 17621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:09.583986 17621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:09.584925 17621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:09.594255 17627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:09.594233 17626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:09.594302 17621 server_base.cc:1034] running on GCE node
W20260620 04:34:09.594257 17629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:09.594803 17621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:09.595383 17621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:09.596560 17621 hybrid_clock.cc:648] HybridClock initialized: now 1781930049596510 us; error 61 us; skew 500 ppm
Jun 20 04:34:09 dist-test-slave-rzqm krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:09.599375 17621 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:34:09.600207 17621 webserver.cc:460] Webserver started at http://127.0.98.2:35567/ using document root <none> and password file <none>
I20260620 04:34:09.600747 17621 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:09.600808 17621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:09.600965 17621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:09.602938 17621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd8fd590b43b46198c7388ede81a9ea7"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "5b9d8f27f6cb1135615ae02f356c32e9"
server_key_iv: "d5337ea2c9dd1f2c2a3386de113d1262"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.603423 17621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd8fd590b43b46198c7388ede81a9ea7"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "5b9d8f27f6cb1135615ae02f356c32e9"
server_key_iv: "d5337ea2c9dd1f2c2a3386de113d1262"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.607100 17621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:34:09.609526 17636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:09.610767 17621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:09.610880 17621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "cd8fd590b43b46198c7388ede81a9ea7"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "5b9d8f27f6cb1135615ae02f356c32e9"
server_key_iv: "d5337ea2c9dd1f2c2a3386de113d1262"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.610985 17621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:09.634383 17621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:09.637372 17621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:09.637547 17621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:09.638195 17621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:09.639065 17621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:09.639112 17621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:09.639164 17621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:09.639190 17621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:09.647331 17621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33283
I20260620 04:34:09.647406 17749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33283 every 8 connection(s)
I20260620 04:34:09.648255 17621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:34:09.651536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 17621
I20260620 04:34:09.651654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:34:09.651942 392 external_mini_cluster.cc:1428] Setting key 71b7a50ddce13b1f4b70ca051f4618c3
Jun 20 04:34:09 dist-test-slave-rzqm krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:34:09.662590 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.650148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60911 (local address 127.0.98.62:41375)
0620 04:34:09.650285 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:09.650288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:09.651181 (+ 893us) server_negotiation.cc:407] Connection header received
0620 04:34:09.652152 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:09.652155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:09.652213 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:09.652290 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:09.653751 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.654315 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.655647 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.655982 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.656083 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.658895 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:09.658900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:09.658903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:09.658932 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:09.660487 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:09.660984 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:09.660988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:09.660990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:09.661119 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:09.661513 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:09.661516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:09.661517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:09.661753 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:09.661858 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:09.662364 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 04:34:09.662466 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:34:09.663450 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.650534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41375 (local address 127.0.98.2:60911)
0620 04:34:09.651023 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:09.651067 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:09.651931 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:09.652467 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:09.652474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:09.652928 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:09.653575 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.653595 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.654438 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.654441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.655509 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.655519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.656136 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.656139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.656292 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.656895 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:09.656911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:09.658702 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:09.660614 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:09.660619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:09.660632 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:09.660886 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:09.661224 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:09.661226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:09.661228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:09.661375 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:09.661854 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:09.661861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:09.662131 (+ 270us) client_negotiation.cc:769] Sending connection context
0620 04:34:09.662308 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 04:34:09.662568 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":118,"threads_started":1}
I20260620 04:34:09.664640 17750 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41375
I20260620 04:34:09.664883 17750 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:09.665396 17750 heartbeater.cc:511] Master 127.0.98.62:41375 requested a full tablet report, sending...
I20260620 04:34:09.666420 17425 ts_manager.cc:194] Registered new tserver with Master: cd8fd590b43b46198c7388ede81a9ea7 (127.0.98.2:33283)
I20260620 04:34:09.667035 17425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:34:09.677454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:09.785965 17755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:09.786268 17755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:09.786331 17755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:09.790510 17755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:09.790584 17755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:09.790684 17755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:34:09.796072 17755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:09.797119 17755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:09.798113 17755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:09.807672 17761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:09.807750 17763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:09.807888 17760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:09.807969 17755 server_base.cc:1034] running on GCE node
I20260620 04:34:09.808387 17755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:09.808993 17755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:09.810213 17755 hybrid_clock.cc:648] HybridClock initialized: now 1781930049810190 us; error 36 us; skew 500 ppm
Jun 20 04:34:09 dist-test-slave-rzqm krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:09.812875 17755 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:34:09.813761 17755 webserver.cc:460] Webserver started at http://127.0.98.3:33337/ using document root <none> and password file <none>
I20260620 04:34:09.814327 17755 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:09.814395 17755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:09.814594 17755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:09.816619 17755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "57ae7f32e1ee44b1b8cf8e38f66491be"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "f3f943a7107ae3275ec4523c7e9bcd7f"
server_key_iv: "6792ab838e724ed0bf5c11608b7f6dbe"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.817170 17755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "57ae7f32e1ee44b1b8cf8e38f66491be"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "f3f943a7107ae3275ec4523c7e9bcd7f"
server_key_iv: "6792ab838e724ed0bf5c11608b7f6dbe"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.820798 17755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 04:34:09.823235 17770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:09.824522 17755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:34:09.824630 17755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "57ae7f32e1ee44b1b8cf8e38f66491be"
format_stamp: "Formatted at 2026-06-20 04:34:09 on dist-test-slave-rzqm"
server_key: "f3f943a7107ae3275ec4523c7e9bcd7f"
server_key_iv: "6792ab838e724ed0bf5c11608b7f6dbe"
server_key_version: "encryptionkey@0"
I20260620 04:34:09.824725 17755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:09.838603 17755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:09.841596 17755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:09.841771 17755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:09.842320 17755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:09.843122 17755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:09.843168 17755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:09.843220 17755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:09.843264 17755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:09.851532 17755 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43395
I20260620 04:34:09.851564 17883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43395 every 8 connection(s)
I20260620 04:34:09.852402 17755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:34:09.853310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 17755
I20260620 04:34:09.853400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:34:09.853642 392 external_mini_cluster.cc:1428] Setting key d9d3698d3a50c90d74ee781654b1e755
Jun 20 04:34:09 dist-test-slave-rzqm krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:09.865316 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.854327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56955 (local address 127.0.98.62:41375)
0620 04:34:09.854448 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:09.854452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:09.855418 (+ 966us) server_negotiation.cc:407] Connection header received
0620 04:34:09.856703 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:09.856705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:09.856786 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:09.856850 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:09.857974 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.858498 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.859558 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.859783 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.859852 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.862173 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:09.862176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:09.862178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:09.862201 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:09.863357 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:09.863840 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:09.863843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:09.863844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:09.863920 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:09.864224 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:09.864227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:09.864229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:09.864412 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:09.864506 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:09.865024 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 04:34:09.865148 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:34:09.866168 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.854731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41375 (local address 127.0.98.3:56955)
0620 04:34:09.855244 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:09.855299 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:09.856505 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:09.856988 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:09.857011 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:09.857372 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:09.857850 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.857864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.858587 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.858589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.859469 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.859478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.859888 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.859891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.860025 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.860585 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:09.860601 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:09.862010 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:09.863470 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:09.863475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:09.863491 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:09.863737 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:09.864001 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:09.864004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:09.864006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:09.864138 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:09.864513 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:09.864519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:09.864768 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 04:34:09.864938 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 04:34:09.865301 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":139,"threads_started":1}
I20260620 04:34:09.867228 17884 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41375
I20260620 04:34:09.867475 17884 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:09.867977 17884 heartbeater.cc:511] Master 127.0.98.62:41375 requested a full tablet report, sending...
I20260620 04:34:09.868981 17425 ts_manager.cc:194] Registered new tserver with Master: 57ae7f32e1ee44b1b8cf8e38f66491be (127.0.98.3:43395)
I20260620 04:34:09.869592 17425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56955
I20260620 04:34:09.879473 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:34:09.887934 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.881248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.0.98.62:41375)
0620 04:34:09.881361 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:09.881365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:09.881503 (+ 138us) server_negotiation.cc:407] Connection header received
0620 04:34:09.881604 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:09.881606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:09.881644 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:09.881708 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:09.882331 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.882833 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.883613 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.883833 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.883913 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.885165 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:09.885168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:09.885170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:09.885195 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:09.886517 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:09.886948 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:09.886951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:09.886952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:09.887026 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:09.887216 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:09.887218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:09.887219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:09.887440 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:09.887539 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:09.887689 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 04:34:09.887781 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:34:09.890542 17425 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:34:09.893553 17425 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:34:09.908119 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.902897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.0.98.1:36585)
0620 04:34:09.903478 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:09.903491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:09.903510 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:34:09.903586 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:09.903591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:09.903778 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:09.903952 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:09.904192 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.905308 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.906716 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.907492 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.907605 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.907767 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:09.907868 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 04:34:09.907963 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":110,"threads_started":1}
I20260620 04:34:09.908114 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.902126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36585 (local address 127.0.0.1:33074)
0620 04:34:09.903114 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:09.903128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:09.903209 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:09.903947 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:09.903949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:09.903967 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:09.904058 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.904064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.905407 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.905411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.906600 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.906610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.907607 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.907610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.907694 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.907725 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 04:34:09.907831 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 04:34:09.907953 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":82,"thread_start_us":93,"threads_started":1}
I20260620 04:34:09.908443 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.902859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.98.3:43395)
0620 04:34:09.903703 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:09.903713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:09.903732 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:34:09.903796 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:09.903800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:09.903999 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:09.904129 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:09.904377 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.905138 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.906949 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.907720 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.907821 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.907952 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:09.908200 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 04:34:09.908298 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":136,"threads_started":1}
I20260620 04:34:09.908697 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.902435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43395 (local address 127.0.0.1:50472)
0620 04:34:09.903256 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:09.903269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:09.903354 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:09.904143 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:09.904146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:09.904163 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:09.904266 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.904273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.905254 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.905258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.906825 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.906836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.907868 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.907871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.907957 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.907976 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:34:09.908042 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 04:34:09.908097 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":134,"thread_start_us":91,"threads_started":1}
I20260620 04:34:09.909384 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.902080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33283 (local address 127.0.0.1:33674)
0620 04:34:09.902801 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:09.902853 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:09.902983 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:09.903709 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:09.903715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:09.903737 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:09.903874 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.903881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.905357 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.905360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.906847 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:09.906854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.908983 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.908986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:09.909098 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.909121 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 04:34:09.909193 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 04:34:09.909264 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":118,"thread_start_us":63,"threads_started":1}
I20260620 04:34:09.909468 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:09.902248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.98.2:33283)
0620 04:34:09.903260 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:09.903274 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:09.903298 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:34:09.903371 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:09.903377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:09.903565 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:09.903711 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:09.903991 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.905226 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.908009 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:09.908874 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:09.908974 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:09.909130 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:09.909242 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 04:34:09.909327 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":144,"threads_started":1}
I20260620 04:34:09.911887 17818 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae6ee2f56664b2588b09a516db80f16 (DEFAULT_TABLE table=test-table [id=4b3ede2fe6634d5598990c83782f445d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:09.911888 17550 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae6ee2f56664b2588b09a516db80f16 (DEFAULT_TABLE table=test-table [id=4b3ede2fe6634d5598990c83782f445d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:09.912309 17684 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae6ee2f56664b2588b09a516db80f16 (DEFAULT_TABLE table=test-table [id=4b3ede2fe6634d5598990c83782f445d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:09.913060 17550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae6ee2f56664b2588b09a516db80f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:09.913060 17818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae6ee2f56664b2588b09a516db80f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:09.913161 17684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae6ee2f56664b2588b09a516db80f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:09.920132 17901 tablet_bootstrap.cc:492] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: Bootstrap starting.
I20260620 04:34:09.920845 17902 tablet_bootstrap.cc:492] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0: Bootstrap starting.
I20260620 04:34:09.920845 17903 tablet_bootstrap.cc:492] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: Bootstrap starting.
I20260620 04:34:09.922194 17901 tablet_bootstrap.cc:654] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:09.922976 17902 tablet_bootstrap.cc:654] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:09.922976 17903 tablet_bootstrap.cc:654] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:09.923203 17901 log.cc:824] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:09.924170 17903 log.cc:824] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:09.924170 17902 log.cc:824] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:09.925376 17901 tablet_bootstrap.cc:492] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: No bootstrap required, opened a new log
I20260620 04:34:09.925618 17901 ts_tablet_manager.cc:1397] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 04:34:09.926304 17903 tablet_bootstrap.cc:492] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: No bootstrap required, opened a new log
I20260620 04:34:09.926487 17902 tablet_bootstrap.cc:492] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0: No bootstrap required, opened a new log
I20260620 04:34:09.926486 17903 ts_tablet_manager.cc:1397] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 04:34:09.926642 17902 ts_tablet_manager.cc:1397] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:34:09.928719 17901 raft_consensus.cc:348] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 } } peers { permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 } } peers { permanent_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43395 } }
I20260620 04:34:09.928956 17901 raft_consensus.cc:374] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:09.929054 17901 raft_consensus.cc:729] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8fd590b43b46198c7388ede81a9ea7, State: Initialized, Role: FOLLOWER
I20260620 04:34:09.929508 17901 consensus_queue.cc:260] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 } } peers { permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 } } peers { permanent_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43395 } }
I20260620 04:34:09.930163 17750 heartbeater.cc:503] Master 127.0.98.62:41375 was elected leader, sending a full tablet report...
I20260620 04:34:09.930482 17901 ts_tablet_manager.cc:1428] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260620 04:34:09.930339 17903 raft_consensus.cc:348] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 } } peers { permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 } } peers { permanent_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43395 } }
I20260620 04:34:09.930660 17903 raft_consensus.cc:374] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:09.930721 17903 raft_consensus.cc:729] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ae7f32e1ee44b1b8cf8e38f66491be, State: Initialized, Role: FOLLOWER
I20260620 04:34:09.931001 17902 raft_consensus.cc:348] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 } } peers { permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 } } peers { permanent_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43395 } }
I20260620 04:34:09.931210 17902 raft_consensus.cc:374] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:09.931145 17903 consensus_queue.cc:260] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 } } peers { permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 } } peers { permanent_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43395 } }
I20260620 04:34:09.931281 17902 raft_consensus.cc:729] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb1737a14ab4f76bf619d4d7bbecfa0, State: Initialized, Role: FOLLOWER
I20260620 04:34:09.931725 17902 consensus_queue.cc:260] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 } } peers { permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 } } peers { permanent_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43395 } }
I20260620 04:34:09.932149 17903 ts_tablet_manager.cc:1428] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 04:34:09.932505 17884 heartbeater.cc:503] Master 127.0.98.62:41375 was elected leader, sending a full tablet report...
I20260620 04:34:09.932742 17902 ts_tablet_manager.cc:1428] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:34:09.933110 17616 heartbeater.cc:503] Master 127.0.98.62:41375 was elected leader, sending a full tablet report...
W20260620 04:34:09.938234 17617 tablet.cc:2369] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:34:10.016259 17908 raft_consensus.cc:482] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:34:10.016477 17908 raft_consensus.cc:504] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 } } peers { permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 } } peers { permanent_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43395 } }
I20260620 04:34:10.017640 17908 leader_election.cc:288] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd8fd590b43b46198c7388ede81a9ea7 (127.0.98.2:33283), cdb1737a14ab4f76bf619d4d7bbecfa0 (127.0.98.1:36585)
I20260620 04:34:10.021219 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.018026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55123 (local address 127.0.98.1:36585)
0620 04:34:10.018149 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.018153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.018218 (+ 65us) server_negotiation.cc:407] Connection header received
0620 04:34:10.018363 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.018366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.018407 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.018469 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:10.018718 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.019206 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.020305 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.020693 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.020783 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.020846 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.021031 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.021098 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:34:10.021231 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.017934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36585 (local address 127.0.98.3:55123)
0620 04:34:10.018124 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:10.018139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:10.018256 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:10.018491 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:10.018494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:10.018509 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:10.018613 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.018619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.019310 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.019313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.020162 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.020194 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.020799 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.020802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.020902 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.020920 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:34:10.020983 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:34:10.021073 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":45}
I20260620 04:34:10.021263 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.018069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37941 (local address 127.0.98.2:33283)
0620 04:34:10.018178 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.018182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.018393 (+ 211us) server_negotiation.cc:407] Connection header received
0620 04:34:10.018531 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.018533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.018572 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.018632 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:10.018874 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.019346 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.020356 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.020756 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.020843 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.020901 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.021082 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.021146 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:34:10.021351 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.017955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33283 (local address 127.0.98.3:37941)
0620 04:34:10.018304 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:10.018315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:10.018426 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:10.018647 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:10.018649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:10.018671 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:10.018784 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.018789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.019442 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.019445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.020240 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.020249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.020857 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.020860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.020956 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.020974 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:34:10.021069 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 04:34:10.021136 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":2,"thread_start_us":124,"threads_started":1}
I20260620 04:34:10.022039 17570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae6ee2f56664b2588b09a516db80f16" candidate_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" is_pre_election: true
I20260620 04:34:10.022282 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae6ee2f56664b2588b09a516db80f16" candidate_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8fd590b43b46198c7388ede81a9ea7" is_pre_election: true
I20260620 04:34:10.022352 17570 raft_consensus.cc:2454] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ae7f32e1ee44b1b8cf8e38f66491be in term 0.
I20260620 04:34:10.022565 17704 raft_consensus.cc:2454] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ae7f32e1ee44b1b8cf8e38f66491be in term 0.
I20260620 04:34:10.022827 17773 leader_election.cc:302] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ae7f32e1ee44b1b8cf8e38f66491be, cdb1737a14ab4f76bf619d4d7bbecfa0; no voters:
I20260620 04:34:10.023137 17908 raft_consensus.cc:2781] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:34:10.023218 17908 raft_consensus.cc:482] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:34:10.023265 17908 raft_consensus.cc:3037] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:10.024389 17908 raft_consensus.cc:504] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 } } peers { permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 } } peers { permanent_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43395 } }
I20260620 04:34:10.024735 17908 leader_election.cc:288] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [CANDIDATE]: Term 1 election: Requested vote from peers cd8fd590b43b46198c7388ede81a9ea7 (127.0.98.2:33283), cdb1737a14ab4f76bf619d4d7bbecfa0 (127.0.98.1:36585)
I20260620 04:34:10.025079 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae6ee2f56664b2588b09a516db80f16" candidate_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8fd590b43b46198c7388ede81a9ea7"
I20260620 04:34:10.025121 17570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae6ee2f56664b2588b09a516db80f16" candidate_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0"
I20260620 04:34:10.025194 17704 raft_consensus.cc:3037] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:10.025243 17570 raft_consensus.cc:3037] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:10.026173 17704 raft_consensus.cc:2454] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ae7f32e1ee44b1b8cf8e38f66491be in term 1.
I20260620 04:34:10.026242 17570 raft_consensus.cc:2454] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ae7f32e1ee44b1b8cf8e38f66491be in term 1.
I20260620 04:34:10.026479 17772 leader_election.cc:302] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ae7f32e1ee44b1b8cf8e38f66491be, cd8fd590b43b46198c7388ede81a9ea7; no voters:
I20260620 04:34:10.026652 17908 raft_consensus.cc:2781] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:10.026858 17908 raft_consensus.cc:686] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 1 LEADER]: Becoming Leader. State: Replica: 57ae7f32e1ee44b1b8cf8e38f66491be, State: Running, Role: LEADER
I20260620 04:34:10.027154 17908 consensus_queue.cc:237] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 } } peers { permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 } } peers { permanent_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43395 } }
I20260620 04:34:10.030288 17421 catalog_manager.cc:5495] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be reported cstate change: term changed from 0 to 1, leader changed from <none> to 57ae7f32e1ee44b1b8cf8e38f66491be (127.0.98.3). New cstate: current_term: 1 leader_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57ae7f32e1ee44b1b8cf8e38f66491be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43395 } health_report { overall_health: HEALTHY } } }
I20260620 04:34:10.062258 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.059132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.0.98.3:43395)
0620 04:34:10.059273 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.059277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.059290 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:34:10.059363 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.059365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.059419 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.059506 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:10.059691 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.060169 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.060965 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.061190 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.061277 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.061506 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:10.061912 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:10.061999 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.062063 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.062113 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:34:10.067917 17570 raft_consensus.cc:1261] T 2ae6ee2f56664b2588b09a516db80f16 P cdb1737a14ab4f76bf619d4d7bbecfa0 [term 1 FOLLOWER]: Refusing update from remote peer 57ae7f32e1ee44b1b8cf8e38f66491be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:10.067917 17704 raft_consensus.cc:1261] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7 [term 1 FOLLOWER]: Refusing update from remote peer 57ae7f32e1ee44b1b8cf8e38f66491be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:10.068637 17911 consensus_queue.cc:1035] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdb1737a14ab4f76bf619d4d7bbecfa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:10.068828 17908 consensus_queue.cc:1035] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8fd590b43b46198c7388ede81a9ea7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:10.076440 17915 mvcc.cc:205] Tried to move back new op lower bound from 7298785485071249408 to 7298785484913741824. Current Snapshot: MvccSnapshot[applied={T|T < 7298785485071249408}]
I20260620 04:34:10.084208 17425 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:10.084362 17425 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:10.086651 17425 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae5f4d2463704fc5853f44fd9aec0f4c: Sending DeleteTablet for 3 replicas of tablet 2ae6ee2f56664b2588b09a516db80f16
I20260620 04:34:10.087575 17818 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae6ee2f56664b2588b09a516db80f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50472
I20260620 04:34:10.087659 17684 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae6ee2f56664b2588b09a516db80f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33674
I20260620 04:34:10.088074 17550 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae6ee2f56664b2588b09a516db80f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33074
I20260620 04:34:10.088181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 17487
I20260620 04:34:10.088431 17926 tablet_replica.cc:330] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: stopping tablet replica
I20260620 04:34:10.088923 17926 raft_consensus.cc:2229] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:34:10.089308 17926 raft_consensus.cc:2258] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:10.089701 17925 tablet_replica.cc:330] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: stopping tablet replica
I20260620 04:34:10.090001 17925 raft_consensus.cc:2229] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:34:10.090407 17925 raft_consensus.cc:2258] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:10.092012 17926 ts_tablet_manager.cc:1905] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:10.094777 17926 ts_tablet_manager.cc:1918] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:10.094877 17926 log.cc:1196] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/2ae6ee2f56664b2588b09a516db80f16
W20260620 04:34:10.095001 17411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36585 (error 108)
I20260620 04:34:10.095176 17926 ts_tablet_manager.cc:1939] T 2ae6ee2f56664b2588b09a516db80f16 P cd8fd590b43b46198c7388ede81a9ea7: Deleting consensus metadata
I20260620 04:34:10.095736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 17621
I20260620 04:34:10.097174 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.096623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36585 (local address 127.0.0.1:33088)
0620 04:34:10.096982 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:10.097081 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269}
W20260620 04:34:10.097360 17411 catalog_manager.cc:4584] TS cdb1737a14ab4f76bf619d4d7bbecfa0 (127.0.98.1:36585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ae6ee2f56664b2588b09a516db80f16: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36585: connect: Connection refused (error 111)
I20260620 04:34:10.098532 17925 ts_tablet_manager.cc:1905] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:10.101193 17925 ts_tablet_manager.cc:1918] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:10.101303 17925 log.cc:1196] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/2ae6ee2f56664b2588b09a516db80f16
I20260620 04:34:10.101679 17925 ts_tablet_manager.cc:1939] T 2ae6ee2f56664b2588b09a516db80f16 P 57ae7f32e1ee44b1b8cf8e38f66491be: Deleting consensus metadata
I20260620 04:34:10.102990 17409 catalog_manager.cc:4841] TS 57ae7f32e1ee44b1b8cf8e38f66491be (127.0.98.3:43395): tablet 2ae6ee2f56664b2588b09a516db80f16 (table test-table [id=4b3ede2fe6634d5598990c83782f445d]) successfully deleted
I20260620 04:34:10.105159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 17755
I20260620 04:34:10.105391 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.105101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33283 (local address 127.0.0.1:33690)
0620 04:34:10.105243 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:10.105303 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 04:34:10.110121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 17393
2026-06-20T04:34:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2201 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T04:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:10Z Disabled control of system clock
I20260620 04:34:10.137966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37027
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:34:10.245184 17931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:10.245466 17931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:10.245500 17931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:10.249518 17931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:10.249586 17931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:10.249600 17931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:10.249615 17931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:10.249629 17931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:10.254422 17931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:10.255502 17931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:10.256871 17931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:10.265311 17937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:10.265488 17931 server_base.cc:1034] running on GCE node
W20260620 04:34:10.265551 17939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:10.265345 17936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:10.266237 17931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:10.267401 17931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:10.268567 17931 hybrid_clock.cc:648] HybridClock initialized: now 1781930050268549 us; error 30 us; skew 500 ppm
I20260620 04:34:10.270413 17931 webserver.cc:460] Webserver started at http://127.0.98.62:34593/ using document root <none> and password file <none>
I20260620 04:34:10.271025 17931 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:10.271086 17931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:10.271271 17931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:10.273396 17931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "1c80384521504e76bdd67ab8f70aad99"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "e973fff76590b238a44684ce23ae2fa1"
server_key_iv: "3eeeb374b52834f0643cdf592bb4df4a"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.273900 17931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c80384521504e76bdd67ab8f70aad99"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "e973fff76590b238a44684ce23ae2fa1"
server_key_iv: "3eeeb374b52834f0643cdf592bb4df4a"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.277535 17931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:10.279980 17945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:10.281376 17931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:10.281492 17931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "1c80384521504e76bdd67ab8f70aad99"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "e973fff76590b238a44684ce23ae2fa1"
server_key_iv: "3eeeb374b52834f0643cdf592bb4df4a"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.281598 17931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:10.290303 17931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:10.290956 17931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:10.291127 17931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:10.297466 17931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37027
I20260620 04:34:10.297497 17997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37027 every 8 connection(s)
I20260620 04:34:10.298374 17931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:10.301457 17998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:10.303169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 17931
I20260620 04:34:10.303282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:10.303542 392 external_mini_cluster.cc:1428] Setting key c359d5dd4fba98128e6caee40984058b
I20260620 04:34:10.307212 17998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99: Bootstrap starting.
I20260620 04:34:10.309378 17998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:10.310077 17998 log.cc:824] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:10.311978 17998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99: No bootstrap required, opened a new log
I20260620 04:34:10.312614 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.304975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.0.98.62:37027)
0620 04:34:10.305644 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.305660 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.305695 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:34:10.306397 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.306417 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.306825 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.307131 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:10.307571 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.308533 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.309428 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.309934 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.310083 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.310938 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:10.310944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:10.310956 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:10.311338 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:10.311392 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:10.311411 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:10.311494 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.311634 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.311882 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":125,"threads_started":1}
I20260620 04:34:10.315713 17998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c80384521504e76bdd67ab8f70aad99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37027 } }
I20260620 04:34:10.315940 17998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:10.315994 17998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c80384521504e76bdd67ab8f70aad99, State: Initialized, Role: FOLLOWER
I20260620 04:34:10.316386 17998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c80384521504e76bdd67ab8f70aad99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37027 } }
I20260620 04:34:10.316514 17998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:10.316576 17998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:10.316671 17998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:10.317584 17998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c80384521504e76bdd67ab8f70aad99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37027 } }
I20260620 04:34:10.317901 17998 leader_election.cc:302] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c80384521504e76bdd67ab8f70aad99; no voters:
I20260620 04:34:10.318181 17998 leader_election.cc:288] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:10.318274 18003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:10.318496 18003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [term 1 LEADER]: Becoming Leader. State: Replica: 1c80384521504e76bdd67ab8f70aad99, State: Running, Role: LEADER
I20260620 04:34:10.318853 18003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c80384521504e76bdd67ab8f70aad99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37027 } }
I20260620 04:34:10.319317 17998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:10.320366 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c80384521504e76bdd67ab8f70aad99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c80384521504e76bdd67ab8f70aad99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37027 } } }
I20260620 04:34:10.320472 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:10.320710 18005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c80384521504e76bdd67ab8f70aad99. Latest consensus state: current_term: 1 leader_uuid: "1c80384521504e76bdd67ab8f70aad99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c80384521504e76bdd67ab8f70aad99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37027 } } }
I20260620 04:34:10.320789 18005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:10.321329 18012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:10.324112 18012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:10.329726 18012 catalog_manager.cc:1347] Generated new cluster ID: 906a359cb25b46279cf70db4effd8ee4
I20260620 04:34:10.329797 18012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:10.338847 18012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:10.340054 18012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:10.348030 18012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99: Generated new TSK 0
I20260620 04:34:10.348726 18012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:10.356117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37027
--builtin_ntp_servers=127.0.98.20:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:34:10.466497 18022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:10.466769 18022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:10.466807 18022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:10.470835 18022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:10.470893 18022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:10.470965 18022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:34:10.475778 18022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:10.476809 18022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:10.478181 18022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:34:10.487855 18022 server_base.cc:1034] running on GCE node
W20260620 04:34:10.487828 18030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:10.487769 18027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:10.487802 18028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:10.488600 18022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:10.489270 18022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:10.490427 18022 hybrid_clock.cc:648] HybridClock initialized: now 1781930050490399 us; error 43 us; skew 500 ppm
I20260620 04:34:10.492184 18022 webserver.cc:460] Webserver started at http://127.0.98.1:37449/ using document root <none> and password file <none>
I20260620 04:34:10.492753 18022 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:10.492820 18022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:10.493029 18022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:10.495052 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "1755134bd9e5432d86c19f4dd6e43046"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "36cdfd1c4fff4e3941dbf274723eb0b3"
server_key_iv: "7e108b5d214a1b3b1ac254b7d41ce2bf"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.495586 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1755134bd9e5432d86c19f4dd6e43046"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "36cdfd1c4fff4e3941dbf274723eb0b3"
server_key_iv: "7e108b5d214a1b3b1ac254b7d41ce2bf"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.499123 18022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:34:10.501510 18036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:10.502732 18022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:10.502836 18022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "1755134bd9e5432d86c19f4dd6e43046"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "36cdfd1c4fff4e3941dbf274723eb0b3"
server_key_iv: "7e108b5d214a1b3b1ac254b7d41ce2bf"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.502935 18022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:10.528052 18022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:10.528776 18022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:10.528951 18022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:10.529577 18022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:10.530530 18022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:10.530586 18022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:10.530654 18022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:10.530678 18022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:10.539223 18022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39809
I20260620 04:34:10.539311 18149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39809 every 8 connection(s)
I20260620 04:34:10.540123 18022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:34:10.542140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 18022
I20260620 04:34:10.542243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:34:10.542555 392 external_mini_cluster.cc:1428] Setting key 1ce7d73665d564136bf1d85e58149a99
I20260620 04:34:10.544416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37027
--builtin_ntp_servers=127.0.98.20:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:34:10.549204 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.541956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59213 (local address 127.0.98.62:37027)
0620 04:34:10.542100 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.542105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.543281 (+ 1176us) server_negotiation.cc:407] Connection header received
0620 04:34:10.544690 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.544693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.544746 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.544827 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:10.545769 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.546299 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.547399 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.547631 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.547724 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.548737 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:10.548741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:10.548744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:10.548757 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:10.548775 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:10.548782 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:10.548838 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.548976 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.549059 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 04:34:10.550011 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.542409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37027 (local address 127.0.98.1:59213)
0620 04:34:10.543138 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:10.543201 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:10.544423 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:10.545037 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:10.545047 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:10.545143 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:10.545632 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.545644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.546416 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.546420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.547283 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.547291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.547775 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.547780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.547932 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.548490 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:34:10.548492 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:10.548531 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:34:10.548592 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:10.548847 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:10.548856 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:10.548862 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:34:10.548966 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 04:34:10.549268 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":122,"threads_started":1}
I20260620 04:34:10.551133 18150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37027
I20260620 04:34:10.551348 18150 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:10.551826 18150 heartbeater.cc:511] Master 127.0.98.62:37027 requested a full tablet report, sending...
I20260620 04:34:10.553259 17962 ts_manager.cc:194] Registered new tserver with Master: 1755134bd9e5432d86c19f4dd6e43046 (127.0.98.1:39809)
I20260620 04:34:10.554348 17962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59213
W20260620 04:34:10.653139 18153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:10.653417 18153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:10.653475 18153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:10.657536 18153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:10.657605 18153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:10.657682 18153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:34:10.662567 18153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:10.663627 18153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:10.664983 18153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:10.674387 18159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:10.674542 18153 server_base.cc:1034] running on GCE node
W20260620 04:34:10.674585 18158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:10.674405 18161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:10.675050 18153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:10.675661 18153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:10.676810 18153 hybrid_clock.cc:648] HybridClock initialized: now 1781930050676791 us; error 33 us; skew 500 ppm
I20260620 04:34:10.678563 18153 webserver.cc:460] Webserver started at http://127.0.98.2:41959/ using document root <none> and password file <none>
I20260620 04:34:10.679100 18153 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:10.679162 18153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:10.679369 18153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:10.681734 18153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "6ee3ae996a18491a93ed371a462199dc"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "5730336f430accec3918c86381d38fc0"
server_key_iv: "8cc3e35458dabdf32f1c98294670d5a9"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.682390 18153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6ee3ae996a18491a93ed371a462199dc"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "5730336f430accec3918c86381d38fc0"
server_key_iv: "8cc3e35458dabdf32f1c98294670d5a9"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.687103 18153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:34:10.689986 18167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:10.691224 18153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:10.691344 18153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "6ee3ae996a18491a93ed371a462199dc"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "5730336f430accec3918c86381d38fc0"
server_key_iv: "8cc3e35458dabdf32f1c98294670d5a9"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.691461 18153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:10.704217 18153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:10.704845 18153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:10.705027 18153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:10.705576 18153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:10.706459 18153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:10.706506 18153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:10.706547 18153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:10.706576 18153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:10.714829 18153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43155
I20260620 04:34:10.714865 18280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43155 every 8 connection(s)
I20260620 04:34:10.715682 18153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:34:10.720343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 18153
I20260620 04:34:10.720495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:34:10.720799 392 external_mini_cluster.cc:1428] Setting key 7d1a19456920e6c61332e249abf9a5ea
I20260620 04:34:10.722918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37027
--builtin_ntp_servers=127.0.98.20:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:34:10.725804 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.717475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44327 (local address 127.0.98.62:37027)
0620 04:34:10.717606 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.717610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.718612 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 04:34:10.719994 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.719997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.720048 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.720123 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:10.721437 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.722147 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.723725 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.723955 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.724040 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.725243 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:10.725253 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:10.725256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:10.725270 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:10.725293 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:10.725303 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:10.725378 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.725596 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.725659 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:34:10.726658 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.717849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37027 (local address 127.0.98.2:44327)
0620 04:34:10.718477 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:10.718516 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:10.719799 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:10.720382 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:10.720392 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:10.720507 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:10.721244 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.721262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.722268 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.722271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.723591 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.723602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.724098 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.724102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.724310 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.724932 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:34:10.724934 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:10.724976 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:34:10.725085 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:10.725411 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:10.725424 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:10.725430 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:34:10.725585 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 04:34:10.725822 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260620 04:34:10.727837 18281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37027
I20260620 04:34:10.728053 18281 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:10.728551 18281 heartbeater.cc:511] Master 127.0.98.62:37027 requested a full tablet report, sending...
I20260620 04:34:10.729641 17962 ts_manager.cc:194] Registered new tserver with Master: 6ee3ae996a18491a93ed371a462199dc (127.0.98.2:43155)
I20260620 04:34:10.730234 17962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44327
W20260620 04:34:10.830788 18284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:10.831035 18284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:10.831077 18284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:10.835028 18284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:10.835083 18284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:10.835141 18284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:34:10.839794 18284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:10.840802 18284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:10.842159 18284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:34:10.851507 18284 server_base.cc:1034] running on GCE node
W20260620 04:34:10.851409 18289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:10.851409 18290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:10.851430 18292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:10.852073 18284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:10.852689 18284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:10.853849 18284 hybrid_clock.cc:648] HybridClock initialized: now 1781930050853814 us; error 46 us; skew 500 ppm
I20260620 04:34:10.855670 18284 webserver.cc:460] Webserver started at http://127.0.98.3:42937/ using document root <none> and password file <none>
I20260620 04:34:10.856230 18284 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:10.856298 18284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:10.856493 18284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:10.858583 18284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "5afa9bdbbfb117b8c7d1db95faa38efa"
server_key_iv: "a52b753147447bea19f3bb880091929b"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.859146 18284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "5afa9bdbbfb117b8c7d1db95faa38efa"
server_key_iv: "a52b753147447bea19f3bb880091929b"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.862849 18284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:34:10.865273 18298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:10.866566 18284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:34:10.866674 18284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a"
format_stamp: "Formatted at 2026-06-20 04:34:10 on dist-test-slave-rzqm"
server_key: "5afa9bdbbfb117b8c7d1db95faa38efa"
server_key_iv: "a52b753147447bea19f3bb880091929b"
server_key_version: "encryptionkey@0"
I20260620 04:34:10.866770 18284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:10.886377 18284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:10.887090 18284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:10.887264 18284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:10.887864 18284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:10.888849 18284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:10.888906 18284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:10.888960 18284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:10.888988 18284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:10.897420 18284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38313
I20260620 04:34:10.897444 18411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38313 every 8 connection(s)
I20260620 04:34:10.898293 18284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:34:10.899214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 18284
I20260620 04:34:10.899348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:34:10.899638 392 external_mini_cluster.cc:1428] Setting key 70d0b1f1959b3d92edfbf1bfd089a4d0
I20260620 04:34:10.907557 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.900398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59747 (local address 127.0.98.62:37027)
0620 04:34:10.900566 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.900571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.901610 (+ 1039us) server_negotiation.cc:407] Connection header received
0620 04:34:10.903076 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.903079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.903132 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.903210 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:10.904091 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.904624 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.905786 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.906063 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.906153 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.907103 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:10.907107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:10.907109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:10.907124 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:10.907145 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:10.907154 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:10.907221 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.907351 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.907413 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 04:34:10.908456 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.900776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37027 (local address 127.0.98.3:59747)
0620 04:34:10.901480 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:10.901520 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:10.902884 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:10.903382 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:10.903389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:10.903474 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:10.903957 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.903971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.904759 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.904761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.905679 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.905688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.906197 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.906201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.906357 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.906867 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:34:10.906868 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:10.906906 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:34:10.906969 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:10.907229 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:10.907238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:10.907244 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:34:10.907346 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 04:34:10.907636 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":154,"threads_started":1}
I20260620 04:34:10.909492 18412 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37027
I20260620 04:34:10.909755 18412 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:10.910246 18412 heartbeater.cc:511] Master 127.0.98.62:37027 requested a full tablet report, sending...
I20260620 04:34:10.911227 17962 ts_manager.cc:194] Registered new tserver with Master: f7b6b9c4b4c8469aa46745ce5fa3fa4a (127.0.98.3:38313)
I20260620 04:34:10.911847 17962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59747
I20260620 04:34:10.913633 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:34:10.918915 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.915458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.0.98.62:37027)
0620 04:34:10.915577 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.915582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.915808 (+ 226us) server_negotiation.cc:407] Connection header received
0620 04:34:10.915872 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.915874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.915921 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.915990 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:10.916243 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.916736 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.917560 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.917807 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.917883 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.918550 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:10.918552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:10.918554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:10.918564 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:10.918582 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:10.918589 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:10.918635 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.918694 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.918739 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:34:10.921417 17962 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:34:10.924294 17962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:34:10.938297 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.933261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.0.98.3:38313)
0620 04:34:10.933700 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.933708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.933728 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:34:10.933794 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.933799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.933977 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.934156 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:10.934385 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.935226 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.937084 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.937631 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.937728 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.937922 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.938042 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.938134 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":125,"threads_started":1}
I20260620 04:34:10.938292 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.932449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38313 (local address 127.0.0.1:45054)
0620 04:34:10.933246 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:10.933270 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:10.933350 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:10.934143 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:10.934146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:10.934167 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:10.934278 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.934285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.935330 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.935333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.936788 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.936798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.937757 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.937761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.937865 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.937902 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 04:34:10.938004 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 04:34:10.938134 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":8,"thread_start_us":96,"threads_started":1}
I20260620 04:34:10.938592 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.933232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39809 (local address 127.0.0.1:39246)
0620 04:34:10.933520 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:10.933532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:10.933812 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:10.934598 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:10.934601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:10.934620 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:10.934705 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.934711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.935867 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.935870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.936840 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.936849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.938062 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.938066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.938262 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.938320 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 04:34:10.938389 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 04:34:10.938456 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":72,"thread_start_us":90,"threads_started":1}
I20260620 04:34:10.938690 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.932499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.0.98.2:43155)
0620 04:34:10.932902 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.932914 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.932940 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:34:10.933175 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.933182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.933401 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.933536 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:10.934246 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.935425 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.936911 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.937730 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.937832 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.938000 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.938449 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.938544 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":124,"threads_started":1}
I20260620 04:34:10.939015 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.932354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43155 (local address 127.0.0.1:52040)
0620 04:34:10.932793 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:10.932837 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:10.932953 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:10.933959 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:10.933965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:10.933986 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:10.934111 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.934118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.935531 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.935535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.936787 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:10.936798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.938062 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.938066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:10.938268 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.938324 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 04:34:10.938842 (+ 518us) client_negotiation.cc:240] Negotiation successful
0620 04:34:10.938907 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":164,"thread_start_us":85,"threads_started":1}
I20260620 04:34:10.939018 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:10.933767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.0.98.1:39809)
0620 04:34:10.934204 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:10.934215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:10.934235 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:34:10.934303 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:10.934308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:10.934467 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:10.934610 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:10.934820 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.935774 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.936967 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:10.937735 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:10.937834 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:10.938000 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:10.938779 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 04:34:10.938874 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":133,"threads_started":1}
I20260620 04:34:10.941833 18215 tablet_service.cc:1467] Processing CreateTablet for tablet ba5b973e923e4e20b32aa501a7d4d7b3 (DEFAULT_TABLE table=test-table [id=9a72d77ddd9a4a09bc8acf457458a4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:10.941851 18346 tablet_service.cc:1467] Processing CreateTablet for tablet ba5b973e923e4e20b32aa501a7d4d7b3 (DEFAULT_TABLE table=test-table [id=9a72d77ddd9a4a09bc8acf457458a4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:10.941851 18084 tablet_service.cc:1467] Processing CreateTablet for tablet ba5b973e923e4e20b32aa501a7d4d7b3 (DEFAULT_TABLE table=test-table [id=9a72d77ddd9a4a09bc8acf457458a4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:10.942782 18215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5b973e923e4e20b32aa501a7d4d7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:10.942849 18084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5b973e923e4e20b32aa501a7d4d7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:10.942849 18346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5b973e923e4e20b32aa501a7d4d7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:10.949695 18431 tablet_bootstrap.cc:492] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: Bootstrap starting.
I20260620 04:34:10.950155 18430 tablet_bootstrap.cc:492] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046: Bootstrap starting.
I20260620 04:34:10.950482 18429 tablet_bootstrap.cc:492] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: Bootstrap starting.
I20260620 04:34:10.952611 18430 tablet_bootstrap.cc:654] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:10.952611 18429 tablet_bootstrap.cc:654] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:10.953577 18430 log.cc:824] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:10.953577 18429 log.cc:824] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:10.955072 18431 tablet_bootstrap.cc:654] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:10.955731 18429 tablet_bootstrap.cc:492] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: No bootstrap required, opened a new log
I20260620 04:34:10.955742 18431 log.cc:824] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:10.955960 18429 ts_tablet_manager.cc:1397] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:34:10.956209 18430 tablet_bootstrap.cc:492] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046: No bootstrap required, opened a new log
I20260620 04:34:10.956419 18430 ts_tablet_manager.cc:1397] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 04:34:10.960318 18430 raft_consensus.cc:348] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 } } peers { permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 } } peers { permanent_uuid: "1755134bd9e5432d86c19f4dd6e43046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39809 } }
I20260620 04:34:10.960319 18429 raft_consensus.cc:348] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 } } peers { permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 } } peers { permanent_uuid: "1755134bd9e5432d86c19f4dd6e43046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39809 } }
I20260620 04:34:10.960608 18429 raft_consensus.cc:374] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:10.960695 18429 raft_consensus.cc:729] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee3ae996a18491a93ed371a462199dc, State: Initialized, Role: FOLLOWER
I20260620 04:34:10.960740 18430 raft_consensus.cc:374] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:10.960786 18430 raft_consensus.cc:729] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1755134bd9e5432d86c19f4dd6e43046, State: Initialized, Role: FOLLOWER
I20260620 04:34:10.961229 18429 consensus_queue.cc:260] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 } } peers { permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 } } peers { permanent_uuid: "1755134bd9e5432d86c19f4dd6e43046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39809 } }
I20260620 04:34:10.961229 18430 consensus_queue.cc:260] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 } } peers { permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 } } peers { permanent_uuid: "1755134bd9e5432d86c19f4dd6e43046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39809 } }
I20260620 04:34:10.961742 18431 tablet_bootstrap.cc:492] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: No bootstrap required, opened a new log
I20260620 04:34:10.961920 18150 heartbeater.cc:503] Master 127.0.98.62:37027 was elected leader, sending a full tablet report...
I20260620 04:34:10.962002 18431 ts_tablet_manager.cc:1397] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 04:34:10.962273 18429 ts_tablet_manager.cc:1428] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:34:10.962574 18281 heartbeater.cc:503] Master 127.0.98.62:37027 was elected leader, sending a full tablet report...
I20260620 04:34:10.963903 18430 ts_tablet_manager.cc:1428] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:34:10.965907 18431 raft_consensus.cc:348] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 } } peers { permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 } } peers { permanent_uuid: "1755134bd9e5432d86c19f4dd6e43046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39809 } }
I20260620 04:34:10.966138 18431 raft_consensus.cc:374] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:10.966184 18431 raft_consensus.cc:729] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b6b9c4b4c8469aa46745ce5fa3fa4a, State: Initialized, Role: FOLLOWER
W20260620 04:34:10.966493 18282 tablet.cc:2369] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:34:10.966641 18431 consensus_queue.cc:260] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 } } peers { permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 } } peers { permanent_uuid: "1755134bd9e5432d86c19f4dd6e43046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39809 } }
I20260620 04:34:10.967398 18412 heartbeater.cc:503] Master 127.0.98.62:37027 was elected leader, sending a full tablet report...
I20260620 04:34:10.967617 18431 ts_tablet_manager.cc:1428] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:34:11.005473 18435 raft_consensus.cc:482] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:34:11.005692 18435 raft_consensus.cc:504] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 } } peers { permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 } } peers { permanent_uuid: "1755134bd9e5432d86c19f4dd6e43046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39809 } }
I20260620 04:34:11.006920 18435 leader_election.cc:288] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ee3ae996a18491a93ed371a462199dc (127.0.98.2:43155), f7b6b9c4b4c8469aa46745ce5fa3fa4a (127.0.98.3:38313)
I20260620 04:34:11.010421 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:11.007130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43155 (local address 127.0.98.1:34857)
0620 04:34:11.007336 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:11.007351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:11.007450 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:11.007722 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:11.007725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:11.007741 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:11.007852 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:11.007857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:11.008570 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:11.008573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:11.009381 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:11.009391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:11.010028 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:11.010031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:11.010123 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:11.010149 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 04:34:11.010212 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:34:11.010283 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 04:34:11.010426 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:11.007243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34857 (local address 127.0.98.2:43155)
0620 04:34:11.007383 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:11.007387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:11.007404 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:34:11.007570 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:11.007573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:11.007624 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:11.007697 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:11.007968 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:11.008469 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:11.009513 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:11.009920 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:11.010011 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:11.010071 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:11.010236 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 04:34:11.010300 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 04:34:11.010619 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:11.007183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38313 (local address 127.0.98.1:41989)
0620 04:34:11.007603 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:11.007613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:11.007705 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:11.007970 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:11.007973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:11.007989 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:11.008116 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:11.008122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:11.008824 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:11.008828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:11.009630 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:11.009639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:11.010229 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:11.010233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:11.010331 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:11.010348 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 04:34:11.010411 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:34:11.010475 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":114,"threads_started":1}
I20260620 04:34:11.010607 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:11.007416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41989 (local address 127.0.98.3:38313)
0620 04:34:11.007541 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:11.007545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:11.007678 (+ 133us) server_negotiation.cc:407] Connection header received
0620 04:34:11.007837 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:11.007840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:11.007878 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:11.007947 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:11.008215 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:11.008715 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:11.009741 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:11.010108 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:11.010197 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:11.010254 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:11.010432 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 04:34:11.010498 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:34:11.011307 18366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5b973e923e4e20b32aa501a7d4d7b3" candidate_uuid: "1755134bd9e5432d86c19f4dd6e43046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" is_pre_election: true
I20260620 04:34:11.011307 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5b973e923e4e20b32aa501a7d4d7b3" candidate_uuid: "1755134bd9e5432d86c19f4dd6e43046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee3ae996a18491a93ed371a462199dc" is_pre_election: true
I20260620 04:34:11.011632 18366 raft_consensus.cc:2454] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1755134bd9e5432d86c19f4dd6e43046 in term 0.
I20260620 04:34:11.011632 18235 raft_consensus.cc:2454] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1755134bd9e5432d86c19f4dd6e43046 in term 0.
I20260620 04:34:11.012149 18037 leader_election.cc:302] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1755134bd9e5432d86c19f4dd6e43046, 6ee3ae996a18491a93ed371a462199dc; no voters:
I20260620 04:34:11.012420 18435 raft_consensus.cc:2781] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:34:11.012499 18435 raft_consensus.cc:482] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:34:11.012550 18435 raft_consensus.cc:3037] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:11.013659 18435 raft_consensus.cc:504] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 } } peers { permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 } } peers { permanent_uuid: "1755134bd9e5432d86c19f4dd6e43046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39809 } }
I20260620 04:34:11.014027 18435 leader_election.cc:288] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [CANDIDATE]: Term 1 election: Requested vote from peers 6ee3ae996a18491a93ed371a462199dc (127.0.98.2:43155), f7b6b9c4b4c8469aa46745ce5fa3fa4a (127.0.98.3:38313)
I20260620 04:34:11.014401 18366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5b973e923e4e20b32aa501a7d4d7b3" candidate_uuid: "1755134bd9e5432d86c19f4dd6e43046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a"
I20260620 04:34:11.014393 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5b973e923e4e20b32aa501a7d4d7b3" candidate_uuid: "1755134bd9e5432d86c19f4dd6e43046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee3ae996a18491a93ed371a462199dc"
I20260620 04:34:11.014524 18235 raft_consensus.cc:3037] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:11.014523 18366 raft_consensus.cc:3037] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:11.015772 18366 raft_consensus.cc:2454] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1755134bd9e5432d86c19f4dd6e43046 in term 1.
I20260620 04:34:11.015769 18235 raft_consensus.cc:2454] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1755134bd9e5432d86c19f4dd6e43046 in term 1.
I20260620 04:34:11.016093 18038 leader_election.cc:302] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1755134bd9e5432d86c19f4dd6e43046, f7b6b9c4b4c8469aa46745ce5fa3fa4a; no voters:
I20260620 04:34:11.016278 18435 raft_consensus.cc:2781] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:11.016474 18435 raft_consensus.cc:686] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [term 1 LEADER]: Becoming Leader. State: Replica: 1755134bd9e5432d86c19f4dd6e43046, State: Running, Role: LEADER
I20260620 04:34:11.016803 18435 consensus_queue.cc:237] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 } } peers { permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 } } peers { permanent_uuid: "1755134bd9e5432d86c19f4dd6e43046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39809 } }
I20260620 04:34:11.020282 17958 catalog_manager.cc:5495] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1755134bd9e5432d86c19f4dd6e43046 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1755134bd9e5432d86c19f4dd6e43046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1755134bd9e5432d86c19f4dd6e43046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39809 } health_report { overall_health: HEALTHY } } }
W20260620 04:34:11.041211 18151 tablet.cc:2369] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:34:11.042572 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:11.039036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39248 (local address 127.0.98.1:39809)
0620 04:34:11.039166 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:11.039170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:11.039185 (+ 15us) server_negotiation.cc:407] Connection header received
0620 04:34:11.039339 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:11.039342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:11.039399 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:11.039488 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:11.039749 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:11.040247 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:11.041094 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:11.041320 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:11.041418 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:11.041765 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:11.042168 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:11.042288 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:11.042367 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 04:34:11.042425 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:34:11.048317 18366 raft_consensus.cc:1261] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a [term 1 FOLLOWER]: Refusing update from remote peer 1755134bd9e5432d86c19f4dd6e43046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:11.048317 18235 raft_consensus.cc:1261] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc [term 1 FOLLOWER]: Refusing update from remote peer 1755134bd9e5432d86c19f4dd6e43046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:11.048923 18439 consensus_queue.cc:1035] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ee3ae996a18491a93ed371a462199dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:11.049126 18435 consensus_queue.cc:1035] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7b6b9c4b4c8469aa46745ce5fa3fa4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:11.057559 18443 mvcc.cc:205] Tried to move back new op lower bound from 7298785489086758912 to 7298785488967696384. Current Snapshot: MvccSnapshot[applied={T|T < 7298785489086758912}]
I20260620 04:34:11.065285 17958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:11.065474 17958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:11.067997 17958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c80384521504e76bdd67ab8f70aad99: Sending DeleteTablet for 3 replicas of tablet ba5b973e923e4e20b32aa501a7d4d7b3
I20260620 04:34:11.068825 18215 tablet_service.cc:1514] Processing DeleteTablet for tablet ba5b973e923e4e20b32aa501a7d4d7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:11 UTC) from {username='slave'} at 127.0.0.1:52040
I20260620 04:34:11.068902 18346 tablet_service.cc:1514] Processing DeleteTablet for tablet ba5b973e923e4e20b32aa501a7d4d7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:11 UTC) from {username='slave'} at 127.0.0.1:45054
I20260620 04:34:11.069023 18084 tablet_service.cc:1514] Processing DeleteTablet for tablet ba5b973e923e4e20b32aa501a7d4d7b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:11 UTC) from {username='slave'} at 127.0.0.1:39246
I20260620 04:34:11.069423 18453 tablet_replica.cc:330] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: stopping tablet replica
I20260620 04:34:11.069640 18455 tablet_replica.cc:330] T ba5b973e923e4e20b32aa501a7d4d7b3 P 1755134bd9e5432d86c19f4dd6e43046: stopping tablet replica
I20260620 04:34:11.069854 18454 tablet_replica.cc:330] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: stopping tablet replica
I20260620 04:34:11.070087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 18022
I20260620 04:34:11.071002 18454 raft_consensus.cc:2229] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:34:11.071331 18454 raft_consensus.cc:2258] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:11.072055 18453 raft_consensus.cc:2229] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:34:11.072400 18453 raft_consensus.cc:2258] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:11.072777 18454 ts_tablet_manager.cc:1905] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:11.073736 18453 ts_tablet_manager.cc:1905] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:11.075043 18454 ts_tablet_manager.cc:1918] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:11.075145 18454 log.cc:1196] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/ba5b973e923e4e20b32aa501a7d4d7b3
I20260620 04:34:11.075469 18454 ts_tablet_manager.cc:1939] T ba5b973e923e4e20b32aa501a7d4d7b3 P f7b6b9c4b4c8469aa46745ce5fa3fa4a: Deleting consensus metadata
I20260620 04:34:11.076324 18453 ts_tablet_manager.cc:1918] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:11.076417 18453 log.cc:1196] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/ba5b973e923e4e20b32aa501a7d4d7b3
I20260620 04:34:11.076589 17947 catalog_manager.cc:4841] TS f7b6b9c4b4c8469aa46745ce5fa3fa4a (127.0.98.3:38313): tablet ba5b973e923e4e20b32aa501a7d4d7b3 (table test-table [id=9a72d77ddd9a4a09bc8acf457458a4ac]) successfully deleted
I20260620 04:34:11.076756 18453 ts_tablet_manager.cc:1939] T ba5b973e923e4e20b32aa501a7d4d7b3 P 6ee3ae996a18491a93ed371a462199dc: Deleting consensus metadata
W20260620 04:34:11.077260 17947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39809 (error 108)
I20260620 04:34:11.077920 17946 catalog_manager.cc:4841] TS 6ee3ae996a18491a93ed371a462199dc (127.0.98.2:43155): tablet ba5b973e923e4e20b32aa501a7d4d7b3 (table test-table [id=9a72d77ddd9a4a09bc8acf457458a4ac]) successfully deleted
I20260620 04:34:11.078056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 18153
I20260620 04:34:11.078318 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:11.078066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39809 (local address 127.0.0.1:39254)
0620 04:34:11.078169 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:11.078232 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 04:34:11.079838 17947 catalog_manager.cc:4584] TS 1755134bd9e5432d86c19f4dd6e43046 (127.0.98.1:39809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba5b973e923e4e20b32aa501a7d4d7b3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39809: connect: Connection refused (error 111)
I20260620 04:34:11.082991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 18284
I20260620 04:34:11.087663 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 17931
2026-06-20T04:34:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (976 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:11 dist-test-slave-rzqm krb5kdc[18459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:11 dist-test-slave-rzqm krb5kdc[18459](info): set up 2 sockets
Jun 20 04:34:11 dist-test-slave-rzqm krb5kdc[18459](info): commencing operation
krb5kdc: starting...
W20260620 04:34:12.044651 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:12 dist-test-slave-rzqm krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:12.158058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41877
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:12.266288 18472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:12.266588 18472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:12.266646 18472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:12.270710 18472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:12.270782 18472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:12.270809 18472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:12.270830 18472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:12.270850 18472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:12.276182 18472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:12.277321 18472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:12.278311 18472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:12.286345 18478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:12.286427 18477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:12.286448 18480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:12.286530 18472 server_base.cc:1034] running on GCE node
I20260620 04:34:12.287197 18472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:12.288280 18472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:12.289435 18472 hybrid_clock.cc:648] HybridClock initialized: now 1781930052289418 us; error 30 us; skew 500 ppm
Jun 20 04:34:12 dist-test-slave-rzqm krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:12.292122 18472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:12.292984 18472 webserver.cc:460] Webserver started at http://127.0.98.62:33043/ using document root <none> and password file <none>
I20260620 04:34:12.293529 18472 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:12.293593 18472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:12.293777 18472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:12.295725 18472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "30d4116a13c6478c91534845e50da707"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "d3f3f3eb6d2e1298f3e8634ba69931a4"
server_key_iv: "3f55777a9c6c236ac8ff31d289d0df77"
server_key_version: "encryptionkey@0"
I20260620 04:34:12.296248 18472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "30d4116a13c6478c91534845e50da707"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "d3f3f3eb6d2e1298f3e8634ba69931a4"
server_key_iv: "3f55777a9c6c236ac8ff31d289d0df77"
server_key_version: "encryptionkey@0"
I20260620 04:34:12.299768 18472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 04:34:12.302127 18487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:12.303315 18472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:12.303423 18472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "30d4116a13c6478c91534845e50da707"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "d3f3f3eb6d2e1298f3e8634ba69931a4"
server_key_iv: "3f55777a9c6c236ac8ff31d289d0df77"
server_key_version: "encryptionkey@0"
I20260620 04:34:12.303519 18472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:12.324563 18472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:12.327407 18472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:12.327589 18472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:12.334514 18472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41877
I20260620 04:34:12.334528 18539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41877 every 8 connection(s)
I20260620 04:34:12.335420 18472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:12.338182 18540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:12.343479 18540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707: Bootstrap starting.
I20260620 04:34:12.344175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 18472
I20260620 04:34:12.344277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:12.344508 392 external_mini_cluster.cc:1428] Setting key f9d9d9c1470438b2d9c249618cb31b8e
I20260620 04:34:12.346129 18540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:12.346997 18540 log.cc:824] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:12.349045 18540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707: No bootstrap required, opened a new log
I20260620 04:34:12.352993 18540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4116a13c6478c91534845e50da707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } }
I20260620 04:34:12.353302 18540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:12.353359 18540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d4116a13c6478c91534845e50da707, State: Initialized, Role: FOLLOWER
Jun 20 04:34:12 dist-test-slave-rzqm krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:12.353794 18540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4116a13c6478c91534845e50da707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } }
I20260620 04:34:12.353924 18540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:12.353991 18540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:12.354087 18540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:12.355149 18540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4116a13c6478c91534845e50da707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } }
I20260620 04:34:12.355504 18540 leader_election.cc:302] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d4116a13c6478c91534845e50da707; no voters:
I20260620 04:34:12.355839 18540 leader_election.cc:288] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:12.355945 18545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:12.356206 18545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [term 1 LEADER]: Becoming Leader. State: Replica: 30d4116a13c6478c91534845e50da707, State: Running, Role: LEADER
I20260620 04:34:12.356554 18545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4116a13c6478c91534845e50da707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } }
I20260620 04:34:12.357204 18540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:12.358283 18547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d4116a13c6478c91534845e50da707. Latest consensus state: current_term: 1 leader_uuid: "30d4116a13c6478c91534845e50da707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4116a13c6478c91534845e50da707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } } }
I20260620 04:34:12.358429 18547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:12.358678 18546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30d4116a13c6478c91534845e50da707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d4116a13c6478c91534845e50da707" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41877 } } }
I20260620 04:34:12.358806 18554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:12.358843 18546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:12.359537 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:12.345715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.0.98.62:41877)
0620 04:34:12.346135 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:12.346158 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:12.346214 (+ 56us) server_negotiation.cc:407] Connection header received
0620 04:34:12.347040 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:12.347064 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:12.347479 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:12.347792 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:12.348417 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:12.349862 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:12.350683 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:12.351391 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:12.351555 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:12.353991 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:12.353998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:12.354015 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:12.354048 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:12.355910 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:12.356404 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:12.356410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:12.356416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:12.356551 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:12.356848 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:12.356851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:12.356853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:12.357393 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:12.357587 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:12.358175 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 04:34:12.358485 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":110,"threads_started":1}
I20260620 04:34:12.362005 18554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:12.367280 18554 catalog_manager.cc:1347] Generated new cluster ID: 6d94deb8011c47c0a93a70420cbf6079
I20260620 04:34:12.367349 18554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:12.383787 18554 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:12.384835 18554 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:12.393827 18554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707: Generated new TSK 0
I20260620 04:34:12.394615 18554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:34:12.427454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41877
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:12.539042 18566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:12.539307 18566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:12.539351 18566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:12.543385 18566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:12.543447 18566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:12.543515 18566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:34:12.548763 18566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:12.549837 18566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:12.550909 18566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:12.560349 18571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:12.560375 18574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:12.560477 18566 server_base.cc:1034] running on GCE node
W20260620 04:34:12.560375 18572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:12.561071 18566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:12.561699 18566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:12.562868 18566 hybrid_clock.cc:648] HybridClock initialized: now 1781930052562832 us; error 35 us; skew 500 ppm
Jun 20 04:34:12 dist-test-slave-rzqm krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:12.565481 18566 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:34:12.566310 18566 webserver.cc:460] Webserver started at http://127.0.98.1:46389/ using document root <none> and password file <none>
I20260620 04:34:12.566814 18566 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:12.566877 18566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:12.567037 18566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:12.569041 18566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "caf5cff2db4a4d1ab57318b9e06c6593"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "86633eae706886ff357a73d72e5b85bd"
server_key_iv: "5c6e681afabece7058711b358b73544d"
server_key_version: "encryptionkey@0"
I20260620 04:34:12.569574 18566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "caf5cff2db4a4d1ab57318b9e06c6593"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "86633eae706886ff357a73d72e5b85bd"
server_key_iv: "5c6e681afabece7058711b358b73544d"
server_key_version: "encryptionkey@0"
I20260620 04:34:12.573248 18566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:12.575687 18581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:12.576992 18566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:34:12.577168 18566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "caf5cff2db4a4d1ab57318b9e06c6593"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "86633eae706886ff357a73d72e5b85bd"
server_key_iv: "5c6e681afabece7058711b358b73544d"
server_key_version: "encryptionkey@0"
I20260620 04:34:12.577272 18566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:12.596627 18566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:12.599592 18566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:12.599767 18566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:12.600378 18566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:12.601253 18566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:12.601305 18566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:12.601343 18566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:12.601380 18566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:12.609673 18566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35575
I20260620 04:34:12.609716 18694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35575 every 8 connection(s)
I20260620 04:34:12.610560 18566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:34:12.613545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 18566
I20260620 04:34:12.613673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:34:12.613962 392 external_mini_cluster.cc:1428] Setting key ac4914845a42acd51f5059fd0471af97
Jun 20 04:34:12 dist-test-slave-rzqm krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:34:12.624425 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:12.612358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56193 (local address 127.0.98.62:41877)
0620 04:34:12.612521 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:12.612526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:12.613407 (+ 881us) server_negotiation.cc:407] Connection header received
0620 04:34:12.614772 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:12.614774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:12.614815 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:12.614909 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:12.616261 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:12.616772 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:12.617947 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:12.618194 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:12.618285 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:12.620893 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:12.620896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:12.620898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:12.620923 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:12.622296 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:12.622820 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:12.622823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:12.622824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:12.622898 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:12.623208 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:12.623212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:12.623213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:12.623468 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:12.623558 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:12.624089 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 04:34:12.624254 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 04:34:12.625398 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:12.612785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41877 (local address 127.0.98.1:56193)
0620 04:34:12.613239 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:12.613298 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:12.614578 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:12.615095 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:12.615104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:12.615568 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:12.616114 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:12.616128 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:12.616897 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:12.616901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:12.617824 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:12.617834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:12.618347 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:12.618351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:12.618511 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:12.619118 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:12.619134 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:12.620717 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:12.622408 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:12.622415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:12.622432 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:12.622725 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:12.622978 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:12.622981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:12.622983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:12.623124 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:12.623568 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:12.623575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:12.623832 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 04:34:12.624006 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 04:34:12.624358 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
I20260620 04:34:12.626650 18695 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41877
I20260620 04:34:12.626919 18695 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:12.627506 18695 heartbeater.cc:511] Master 127.0.98.62:41877 requested a full tablet report, sending...
I20260620 04:34:12.628983 18499 ts_manager.cc:194] Registered new tserver with Master: caf5cff2db4a4d1ab57318b9e06c6593 (127.0.98.1:35575)
I20260620 04:34:12.630162 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:34:12.638164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41877
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:12.745684 18700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:12.745954 18700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:12.745998 18700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:12.749972 18700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:12.750036 18700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:12.750106 18700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:34:12.755401 18700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:12.756414 18700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:12.757465 18700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:12.766690 18706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:12.766690 18705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:12.766762 18708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:12.766944 18700 server_base.cc:1034] running on GCE node
I20260620 04:34:12.767329 18700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:12.767918 18700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:12.769064 18700 hybrid_clock.cc:648] HybridClock initialized: now 1781930052769038 us; error 41 us; skew 500 ppm
Jun 20 04:34:12 dist-test-slave-rzqm krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:12.771701 18700 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:34:12.772553 18700 webserver.cc:460] Webserver started at http://127.0.98.2:39509/ using document root <none> and password file <none>
I20260620 04:34:12.773111 18700 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:12.773183 18700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:12.773382 18700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:12.775346 18700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "8666a3a5cf9941f6bade7a78317bc203"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "0d8f136631fc358f9d801905f6f7a9cc"
server_key_iv: "ff01fe69377a65597a9edaf301d4ef9f"
server_key_version: "encryptionkey@0"
I20260620 04:34:12.775863 18700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8666a3a5cf9941f6bade7a78317bc203"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "0d8f136631fc358f9d801905f6f7a9cc"
server_key_iv: "ff01fe69377a65597a9edaf301d4ef9f"
server_key_version: "encryptionkey@0"
I20260620 04:34:12.779392 18700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:34:12.781770 18715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:12.782951 18700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:12.783062 18700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "8666a3a5cf9941f6bade7a78317bc203"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "0d8f136631fc358f9d801905f6f7a9cc"
server_key_iv: "ff01fe69377a65597a9edaf301d4ef9f"
server_key_version: "encryptionkey@0"
I20260620 04:34:12.783151 18700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:12.819000 18700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:12.822335 18700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:12.822530 18700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:12.823196 18700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:12.824055 18700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:12.824103 18700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:12.824156 18700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:12.824185 18700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:12.832760 18700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43097
I20260620 04:34:12.832891 18828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43097 every 8 connection(s)
I20260620 04:34:12.833702 18700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:34:12.834319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 18700
I20260620 04:34:12.834437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:34:12.834729 392 external_mini_cluster.cc:1428] Setting key 27a5394c1bd61fa5b7aa332fdcdd83e6
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Jun 20 04:34:12 dist-test-slave-rzqm krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:34:12.847664 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:12.835622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40237 (local address 127.0.98.62:41877)
0620 04:34:12.835740 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:12.835744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:12.836861 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 04:34:12.838104 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:12.838107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:12.838160 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:12.838247 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:12.839368 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:12.840151 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:12.841338 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:12.841675 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:12.841785 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:12.844213 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:12.844216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:12.844218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:12.844245 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:12.845491 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:12.846010 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:12.846014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:12.846016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:12.846106 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:12.846468 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:12.846470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:12.846471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:12.846640 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:12.846709 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:12.847359 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 04:34:12.847498 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:34:12.848470 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:12.836183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41877 (local address 127.0.98.2:40237)
0620 04:34:12.836683 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:12.836735 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:12.837904 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:12.838399 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:12.838406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:12.838758 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:12.839225 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:12.839238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:12.840263 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:12.840266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:12.841223 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:12.841232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:12.841809 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:12.841811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:12.841957 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:12.842551 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:12.842567 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:12.844052 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:12.845606 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:12.845612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:12.845625 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:12.845917 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:12.846196 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:12.846199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:12.846201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:12.846385 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:12.846720 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:12.846727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:12.847077 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 04:34:12.847289 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 04:34:12.847616 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":122,"threads_started":1}
I20260620 04:34:12.849617 18829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41877
I20260620 04:34:12.849901 18829 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:12.850498 18829 heartbeater.cc:511] Master 127.0.98.62:41877 requested a full tablet report, sending...
I20260620 04:34:12.851663 18499 ts_manager.cc:194] Registered new tserver with Master: 8666a3a5cf9941f6bade7a78317bc203 (127.0.98.2:43097)
I20260620 04:34:12.852427 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:34:12.861296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41877
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:12.969626 18834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:12.969950 18834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:12.970016 18834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:12.974231 18834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:12.974300 18834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:12.974381 18834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:34:12.979907 18834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:12.981123 18834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:12.982220 18834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:12.992113 18839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:12.992120 18840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:12.992208 18834 server_base.cc:1034] running on GCE node
W20260620 04:34:12.992136 18842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:12.992808 18834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:12.993449 18834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:12.994644 18834 hybrid_clock.cc:648] HybridClock initialized: now 1781930052994590 us; error 63 us; skew 500 ppm
Jun 20 04:34:12 dist-test-slave-rzqm krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:12.997450 18834 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:34:12.998309 18834 webserver.cc:460] Webserver started at http://127.0.98.3:33235/ using document root <none> and password file <none>
I20260620 04:34:12.998871 18834 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:12.998939 18834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:12.999128 18834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:13.001225 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "b8e609397df348eaa740774e58548712"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "bb0d6aa163745d68503490efb9e24332"
server_key_iv: "b4583c15f0751ab7cf0322258b3ad51e"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.001770 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b8e609397df348eaa740774e58548712"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "bb0d6aa163745d68503490efb9e24332"
server_key_iv: "b4583c15f0751ab7cf0322258b3ad51e"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.005491 18834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:13.008123 18849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:13.009521 18834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:13.009642 18834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "b8e609397df348eaa740774e58548712"
format_stamp: "Formatted at 2026-06-20 04:34:12 on dist-test-slave-rzqm"
server_key: "bb0d6aa163745d68503490efb9e24332"
server_key_iv: "b4583c15f0751ab7cf0322258b3ad51e"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.009748 18834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:13.023032 18834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:13.026072 18834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:13.026258 18834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:13.026870 18834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:13.027778 18834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:13.027828 18834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:13.027882 18834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:13.027916 18834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:13.036343 18834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43885
I20260620 04:34:13.036381 18962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43885 every 8 connection(s)
I20260620 04:34:13.037292 18834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 04:34:13 dist-test-slave-rzqm krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:13.047379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 18834
I20260620 04:34:13.047508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:34:13.047775 392 external_mini_cluster.cc:1428] Setting key 9127408b495e77427a1ebac593c86918
I20260620 04:34:13.050746 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.039052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34251 (local address 127.0.98.62:41877)
0620 04:34:13.039216 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.039221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.040064 (+ 843us) server_negotiation.cc:407] Connection header received
0620 04:34:13.041385 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.041388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.041436 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.041511 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:13.042674 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.043202 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.044288 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.044505 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.044580 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.046975 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:13.046978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:13.046980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:13.047005 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:13.048479 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:13.049063 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:13.049067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:13.049068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:13.049167 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:13.049574 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:13.049577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:13.049579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:13.049846 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:13.049940 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.050440 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.050570 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:34:13.051601 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.039448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41877 (local address 127.0.98.3:34251)
0620 04:34:13.039912 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:13.039955 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:13.041169 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:13.041681 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:13.041688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:13.042045 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:13.042549 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.042562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.043315 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.043317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.044192 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.044201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.044656 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.044659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.044805 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.045375 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:13.045391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:13.046814 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:13.048585 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:13.048592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:13.048609 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:13.048915 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:13.049275 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:13.049278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:13.049279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:13.049465 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:13.049949 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:13.049956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:13.050194 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 04:34:13.050380 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 04:34:13.050716 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":98,"threads_started":1}
I20260620 04:34:13.052614 18963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41877
I20260620 04:34:13.052841 18963 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:13.053340 18963 heartbeater.cc:511] Master 127.0.98.62:41877 requested a full tablet report, sending...
I20260620 04:34:13.054262 18499 ts_manager.cc:194] Registered new tserver with Master: b8e609397df348eaa740774e58548712 (127.0.98.3:43885)
I20260620 04:34:13.054795 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34251
I20260620 04:34:13.061628 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:34:13.070367 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.063487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.0.98.62:41877)
0620 04:34:13.063620 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.063624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.063754 (+ 130us) server_negotiation.cc:407] Connection header received
0620 04:34:13.063854 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.063856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.063896 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.063958 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:13.064629 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.065192 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.065993 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.066201 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.066296 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.067672 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:13.067676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:13.067678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:13.067706 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:13.068859 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:13.069336 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:13.069338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:13.069340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:13.069412 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:13.069645 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:13.069647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:13.069648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:13.069834 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:13.069939 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.070116 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.070188 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 04:34:13.073190 18499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:34:13.076095 18499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:34:13.090255 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.084008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.0.98.1:35575)
0620 04:34:13.084312 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.084330 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.084818 (+ 488us) server_negotiation.cc:407] Connection header received
0620 04:34:13.085082 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.085089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.085302 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.085434 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:13.086044 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.087147 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.088762 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.089531 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.089639 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.089787 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.089990 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.090091 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20260620 04:34:13.090266 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.083881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35575 (local address 127.0.0.1:58770)
0620 04:34:13.084680 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:13.084733 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:13.084904 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:13.085652 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:13.085658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:13.085687 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:13.085897 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.085906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.087274 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.087277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.088640 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.088649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.089654 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.089658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.089762 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.089801 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 04:34:13.089950 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 04:34:13.090082 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":639,"spinlock_wait_cycles":3840,"thread_start_us":110,"threads_started":1}
I20260620 04:34:13.090292 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.083881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43097 (local address 127.0.0.1:45436)
0620 04:34:13.084732 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:13.084747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:13.084878 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:13.085656 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:13.085659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:13.085692 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:13.085848 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.085856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.087312 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.087315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.088905 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.088916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.089870 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.089873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.089959 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.089983 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 04:34:13.090044 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 04:34:13.090118 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":28,"spinlock_wait_cycles":896,"thread_start_us":82,"threads_started":1}
I20260620 04:34:13.090530 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.084648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43885 (local address 127.0.0.1:44468)
0620 04:34:13.085587 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:13.085600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:13.085668 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:13.086295 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:13.086298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:13.086315 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:13.086409 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.086415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.087472 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.087475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.088982 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.088990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.090181 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.090184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.090266 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.090287 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:34:13.090352 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 04:34:13.090417 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":78,"thread_start_us":80,"threads_started":1}
I20260620 04:34:13.090902 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.085183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.0.98.3:43885)
0620 04:34:13.085935 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.085946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.085966 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:34:13.086038 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.086042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.086180 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.086306 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:13.086513 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.087337 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.089139 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.089902 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.089997 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.090123 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.090660 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.090758 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":157,"threads_started":1}
I20260620 04:34:13.091042 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.084277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.0.98.2:43097)
0620 04:34:13.085172 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.085183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.085206 (+ 23us) server_negotiation.cc:407] Connection header received
0620 04:34:13.085279 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.085285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.085527 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.085675 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:13.085957 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.087180 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.089062 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.089623 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.089721 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.089896 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.090611 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.090730 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":107,"threads_started":1}
I20260620 04:34:13.093608 18763 tablet_service.cc:1467] Processing CreateTablet for tablet e86fd7b578fd4cfdbe2faa943dbd28b6 (DEFAULT_TABLE table=test-table [id=4085fee3dfbb4c0ab95b6642c78ea0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:13.093734 18629 tablet_service.cc:1467] Processing CreateTablet for tablet e86fd7b578fd4cfdbe2faa943dbd28b6 (DEFAULT_TABLE table=test-table [id=4085fee3dfbb4c0ab95b6642c78ea0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:13.093734 18897 tablet_service.cc:1467] Processing CreateTablet for tablet e86fd7b578fd4cfdbe2faa943dbd28b6 (DEFAULT_TABLE table=test-table [id=4085fee3dfbb4c0ab95b6642c78ea0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:13.094581 18763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86fd7b578fd4cfdbe2faa943dbd28b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:13.094729 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86fd7b578fd4cfdbe2faa943dbd28b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:13.095546 18629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86fd7b578fd4cfdbe2faa943dbd28b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:13.100517 18980 tablet_bootstrap.cc:492] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: Bootstrap starting.
I20260620 04:34:13.101943 18981 tablet_bootstrap.cc:492] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: Bootstrap starting.
I20260620 04:34:13.102891 18980 tablet_bootstrap.cc:654] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:13.103006 18982 tablet_bootstrap.cc:492] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593: Bootstrap starting.
I20260620 04:34:13.104126 18981 tablet_bootstrap.cc:654] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:13.104205 18980 log.cc:824] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:13.105104 18982 tablet_bootstrap.cc:654] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:13.105226 18981 log.cc:824] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:13.106223 18982 log.cc:824] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:13.106396 18980 tablet_bootstrap.cc:492] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: No bootstrap required, opened a new log
I20260620 04:34:13.106593 18980 ts_tablet_manager.cc:1397] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:34:13.107126 18981 tablet_bootstrap.cc:492] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: No bootstrap required, opened a new log
I20260620 04:34:13.107304 18981 ts_tablet_manager.cc:1397] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 04:34:13.107968 18982 tablet_bootstrap.cc:492] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593: No bootstrap required, opened a new log
I20260620 04:34:13.108139 18982 ts_tablet_manager.cc:1397] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:34:13.110461 18980 raft_consensus.cc:348] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "8666a3a5cf9941f6bade7a78317bc203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43097 } } peers { permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } }
I20260620 04:34:13.110769 18980 raft_consensus.cc:374] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:13.110862 18980 raft_consensus.cc:729] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8666a3a5cf9941f6bade7a78317bc203, State: Initialized, Role: FOLLOWER
I20260620 04:34:13.110867 18982 raft_consensus.cc:348] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "8666a3a5cf9941f6bade7a78317bc203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43097 } } peers { permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } }
I20260620 04:34:13.111148 18982 raft_consensus.cc:374] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:13.110982 18981 raft_consensus.cc:348] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "8666a3a5cf9941f6bade7a78317bc203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43097 } } peers { permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } }
I20260620 04:34:13.111209 18982 raft_consensus.cc:729] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf5cff2db4a4d1ab57318b9e06c6593, State: Initialized, Role: FOLLOWER
I20260620 04:34:13.111222 18981 raft_consensus.cc:374] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:13.111264 18981 raft_consensus.cc:729] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e609397df348eaa740774e58548712, State: Initialized, Role: FOLLOWER
I20260620 04:34:13.111326 18980 consensus_queue.cc:260] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "8666a3a5cf9941f6bade7a78317bc203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43097 } } peers { permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } }
I20260620 04:34:13.111644 18982 consensus_queue.cc:260] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "8666a3a5cf9941f6bade7a78317bc203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43097 } } peers { permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } }
I20260620 04:34:13.111714 18981 consensus_queue.cc:260] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "8666a3a5cf9941f6bade7a78317bc203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43097 } } peers { permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } }
I20260620 04:34:13.112123 18829 heartbeater.cc:503] Master 127.0.98.62:41877 was elected leader, sending a full tablet report...
I20260620 04:34:13.112452 18980 ts_tablet_manager.cc:1428] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:34:13.112623 18981 ts_tablet_manager.cc:1428] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 04:34:13.112890 18963 heartbeater.cc:503] Master 127.0.98.62:41877 was elected leader, sending a full tablet report...
I20260620 04:34:13.113799 18982 ts_tablet_manager.cc:1428] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:34:13.114037 18695 heartbeater.cc:503] Master 127.0.98.62:41877 was elected leader, sending a full tablet report...
I20260620 04:34:13.195842 18986 raft_consensus.cc:482] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:34:13.196080 18986 raft_consensus.cc:504] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "8666a3a5cf9941f6bade7a78317bc203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43097 } } peers { permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } }
I20260620 04:34:13.197377 18986 leader_election.cc:288] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caf5cff2db4a4d1ab57318b9e06c6593 (127.0.98.1:35575), b8e609397df348eaa740774e58548712 (127.0.98.3:43885)
I20260620 04:34:13.200870 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.197717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58367 (local address 127.0.98.1:35575)
0620 04:34:13.197846 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.197851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.197871 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:34:13.197991 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.197994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.198044 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.198132 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:13.198420 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.198980 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.199987 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.200375 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.200475 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.200539 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.200695 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.200772 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:34:13.200870 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.197635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35575 (local address 127.0.98.2:58367)
0620 04:34:13.197757 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:13.197772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:13.197870 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:13.198152 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:13.198155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:13.198174 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:13.198313 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.198319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.199091 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.199094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.199866 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.199874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.200482 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.200485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.200586 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.200604 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:34:13.200668 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 04:34:13.200739 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 04:34:13.201272 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.198040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51649 (local address 127.0.98.3:43885)
0620 04:34:13.198173 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.198177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.198371 (+ 194us) server_negotiation.cc:407] Connection header received
0620 04:34:13.198478 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.198481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.198529 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.198595 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:13.198838 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.199321 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.200301 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.200711 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.200797 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.200863 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.201078 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.201149 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:34:13.201597 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.197897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43885 (local address 127.0.98.2:51649)
0620 04:34:13.198297 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:13.198308 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:13.198394 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:13.198614 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:13.198617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:13.198631 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:13.198744 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.198749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.199425 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.199428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.200183 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.200200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.200819 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.200823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.200917 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.200938 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:34:13.201413 (+ 475us) client_negotiation.cc:240] Negotiation successful
0620 04:34:13.201489 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":135,"threads_started":1}
I20260620 04:34:13.201535 18649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86fd7b578fd4cfdbe2faa943dbd28b6" candidate_uuid: "8666a3a5cf9941f6bade7a78317bc203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" is_pre_election: true
I20260620 04:34:13.201844 18649 raft_consensus.cc:2454] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8666a3a5cf9941f6bade7a78317bc203 in term 0.
I20260620 04:34:13.202210 18917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86fd7b578fd4cfdbe2faa943dbd28b6" candidate_uuid: "8666a3a5cf9941f6bade7a78317bc203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e609397df348eaa740774e58548712" is_pre_election: true
I20260620 04:34:13.202327 18717 leader_election.cc:302] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8666a3a5cf9941f6bade7a78317bc203, caf5cff2db4a4d1ab57318b9e06c6593; no voters:
I20260620 04:34:13.202523 18917 raft_consensus.cc:2454] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8666a3a5cf9941f6bade7a78317bc203 in term 0.
I20260620 04:34:13.202597 18986 raft_consensus.cc:2781] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:34:13.202695 18986 raft_consensus.cc:482] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:34:13.202749 18986 raft_consensus.cc:3037] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:13.203758 18986 raft_consensus.cc:504] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "8666a3a5cf9941f6bade7a78317bc203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43097 } } peers { permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } }
I20260620 04:34:13.204087 18986 leader_election.cc:288] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [CANDIDATE]: Term 1 election: Requested vote from peers caf5cff2db4a4d1ab57318b9e06c6593 (127.0.98.1:35575), b8e609397df348eaa740774e58548712 (127.0.98.3:43885)
I20260620 04:34:13.204487 18917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86fd7b578fd4cfdbe2faa943dbd28b6" candidate_uuid: "8666a3a5cf9941f6bade7a78317bc203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e609397df348eaa740774e58548712"
I20260620 04:34:13.204526 18649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86fd7b578fd4cfdbe2faa943dbd28b6" candidate_uuid: "8666a3a5cf9941f6bade7a78317bc203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf5cff2db4a4d1ab57318b9e06c6593"
I20260620 04:34:13.204612 18917 raft_consensus.cc:3037] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:13.204646 18649 raft_consensus.cc:3037] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:13.205699 18649 raft_consensus.cc:2454] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8666a3a5cf9941f6bade7a78317bc203 in term 1.
I20260620 04:34:13.205714 18917 raft_consensus.cc:2454] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8666a3a5cf9941f6bade7a78317bc203 in term 1.
I20260620 04:34:13.205973 18717 leader_election.cc:302] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8666a3a5cf9941f6bade7a78317bc203, b8e609397df348eaa740774e58548712; no voters:
I20260620 04:34:13.206116 18986 raft_consensus.cc:2781] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:13.206326 18986 raft_consensus.cc:686] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 1 LEADER]: Becoming Leader. State: Replica: 8666a3a5cf9941f6bade7a78317bc203, State: Running, Role: LEADER
I20260620 04:34:13.206664 18986 consensus_queue.cc:237] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "8666a3a5cf9941f6bade7a78317bc203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43097 } } peers { permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } }
I20260620 04:34:13.209753 18498 catalog_manager.cc:5495] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8666a3a5cf9941f6bade7a78317bc203 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8666a3a5cf9941f6bade7a78317bc203" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8666a3a5cf9941f6bade7a78317bc203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } health_report { overall_health: UNKNOWN } } }
I20260620 04:34:13.242933 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.239698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.0.98.2:43097)
0620 04:34:13.239822 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.239826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.239839 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:34:13.239944 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.239946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.239999 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.240087 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:13.240284 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.240764 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.241617 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.241828 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.241919 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.242148 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:13.242560 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:13.242677 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.242753 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.242787 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:34:13.248850 18649 raft_consensus.cc:1261] T e86fd7b578fd4cfdbe2faa943dbd28b6 P caf5cff2db4a4d1ab57318b9e06c6593 [term 1 FOLLOWER]: Refusing update from remote peer 8666a3a5cf9941f6bade7a78317bc203: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:13.248850 18917 raft_consensus.cc:1261] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712 [term 1 FOLLOWER]: Refusing update from remote peer 8666a3a5cf9941f6bade7a78317bc203: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:13.249542 18986 consensus_queue.cc:1035] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8e609397df348eaa740774e58548712" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:13.249745 18990 consensus_queue.cc:1035] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf5cff2db4a4d1ab57318b9e06c6593" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:13.266115 18498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:13.266294 18498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:13.268603 18498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30d4116a13c6478c91534845e50da707: Sending DeleteTablet for 3 replicas of tablet e86fd7b578fd4cfdbe2faa943dbd28b6
I20260620 04:34:13.269521 18897 tablet_service.cc:1514] Processing DeleteTablet for tablet e86fd7b578fd4cfdbe2faa943dbd28b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44468
I20260620 04:34:13.269583 18763 tablet_service.cc:1514] Processing DeleteTablet for tablet e86fd7b578fd4cfdbe2faa943dbd28b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45436
I20260620 04:34:13.270090 18629 tablet_service.cc:1514] Processing DeleteTablet for tablet e86fd7b578fd4cfdbe2faa943dbd28b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58770
I20260620 04:34:13.270251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 18566
I20260620 04:34:13.271030 19004 tablet_replica.cc:330] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: stopping tablet replica
I20260620 04:34:13.271416 19004 raft_consensus.cc:2229] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:34:13.272097 19004 raft_consensus.cc:2258] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:13.273680 19003 tablet_replica.cc:330] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: stopping tablet replica
I20260620 04:34:13.274061 19003 raft_consensus.cc:2229] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:34:13.274286 19004 ts_tablet_manager.cc:1905] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:13.274444 19003 raft_consensus.cc:2258] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:13.276041 19003 ts_tablet_manager.cc:1905] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:13.277143 19004 ts_tablet_manager.cc:1918] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:13.277294 19004 log.cc:1196] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/e86fd7b578fd4cfdbe2faa943dbd28b6
W20260620 04:34:13.277846 18489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35575 (error 108)
I20260620 04:34:13.277890 19004 ts_tablet_manager.cc:1939] T e86fd7b578fd4cfdbe2faa943dbd28b6 P 8666a3a5cf9941f6bade7a78317bc203: Deleting consensus metadata
I20260620 04:34:13.278786 19003 ts_tablet_manager.cc:1918] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:13.278926 19003 log.cc:1196] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/e86fd7b578fd4cfdbe2faa943dbd28b6
I20260620 04:34:13.279099 18490 catalog_manager.cc:4841] TS 8666a3a5cf9941f6bade7a78317bc203 (127.0.98.2:43097): tablet e86fd7b578fd4cfdbe2faa943dbd28b6 (table test-table [id=4085fee3dfbb4c0ab95b6642c78ea0ee]) successfully deleted
I20260620 04:34:13.279268 19003 ts_tablet_manager.cc:1939] T e86fd7b578fd4cfdbe2faa943dbd28b6 P b8e609397df348eaa740774e58548712: Deleting consensus metadata
I20260620 04:34:13.279279 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.279010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35575 (local address 127.0.0.1:58782)
0620 04:34:13.279124 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:13.279190 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 04:34:13.279305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 18700
W20260620 04:34:13.280249 18489 catalog_manager.cc:4584] TS caf5cff2db4a4d1ab57318b9e06c6593 (127.0.98.1:35575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e86fd7b578fd4cfdbe2faa943dbd28b6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35575: connect: Connection refused (error 111)
I20260620 04:34:13.281996 18489 catalog_manager.cc:4841] TS b8e609397df348eaa740774e58548712 (127.0.98.3:43885): tablet e86fd7b578fd4cfdbe2faa943dbd28b6 (table test-table [id=4085fee3dfbb4c0ab95b6642c78ea0ee]) successfully deleted
I20260620 04:34:13.284775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 18834
I20260620 04:34:13.289342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 18472
2026-06-20T04:34:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2202 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T04:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:13Z Disabled control of system clock
I20260620 04:34:13.316644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40699
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:34:13.424566 19009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:13.424846 19009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:13.424889 19009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:13.428884 19009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:13.428952 19009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:13.428966 19009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:13.428982 19009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:13.429014 19009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:13.433892 19009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:13.434921 19009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:13.436233 19009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:13.444411 19015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:13.444429 19014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:13.444548 19009 server_base.cc:1034] running on GCE node
W20260620 04:34:13.444463 19017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:13.445259 19009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:13.446417 19009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:13.447594 19009 hybrid_clock.cc:648] HybridClock initialized: now 1781930053447577 us; error 29 us; skew 500 ppm
I20260620 04:34:13.449462 19009 webserver.cc:460] Webserver started at http://127.0.98.62:35501/ using document root <none> and password file <none>
I20260620 04:34:13.450009 19009 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:13.450078 19009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:13.450284 19009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:13.452378 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "e02ec1636ea64456a0137253cbc03745"
format_stamp: "Formatted at 2026-06-20 04:34:13 on dist-test-slave-rzqm"
server_key: "e37f7502f33c0907fc07fa3ebe9a36e9"
server_key_iv: "66a9b04775c7300b7e83563c5293302c"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.452935 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "e02ec1636ea64456a0137253cbc03745"
format_stamp: "Formatted at 2026-06-20 04:34:13 on dist-test-slave-rzqm"
server_key: "e37f7502f33c0907fc07fa3ebe9a36e9"
server_key_iv: "66a9b04775c7300b7e83563c5293302c"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.456646 19009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:34:13.459169 19023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:13.460445 19009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:34:13.460574 19009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "e02ec1636ea64456a0137253cbc03745"
format_stamp: "Formatted at 2026-06-20 04:34:13 on dist-test-slave-rzqm"
server_key: "e37f7502f33c0907fc07fa3ebe9a36e9"
server_key_iv: "66a9b04775c7300b7e83563c5293302c"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.460690 19009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:13.478390 19009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:13.479121 19009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:13.479314 19009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:13.485749 19009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40699
I20260620 04:34:13.485764 19075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40699 every 8 connection(s)
I20260620 04:34:13.486692 19009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:13.489791 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:13.492017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 19009
I20260620 04:34:13.492136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:13.492409 392 external_mini_cluster.cc:1428] Setting key c9555f28d916232dd62dd01494b01cc3
I20260620 04:34:13.495639 19076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745: Bootstrap starting.
I20260620 04:34:13.498108 19076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:13.498878 19076 log.cc:824] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:13.500954 19076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745: No bootstrap required, opened a new log
I20260620 04:34:13.501595 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.493755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.0.98.62:40699)
0620 04:34:13.494379 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.494394 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.494429 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:34:13.495059 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.495076 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.495475 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.495774 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:13.496191 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.497199 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.498337 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.498801 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.498927 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.499828 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:13.499834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:13.499847 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:13.500219 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:13.500273 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:13.500304 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:13.500377 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.500549 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.500813 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":129,"threads_started":1}
I20260620 04:34:13.504730 19076 raft_consensus.cc:348] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02ec1636ea64456a0137253cbc03745" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } }
I20260620 04:34:13.504958 19076 raft_consensus.cc:374] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:13.505023 19076 raft_consensus.cc:729] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02ec1636ea64456a0137253cbc03745, State: Initialized, Role: FOLLOWER
I20260620 04:34:13.505409 19076 consensus_queue.cc:260] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02ec1636ea64456a0137253cbc03745" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } }
I20260620 04:34:13.505529 19076 raft_consensus.cc:388] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:13.505589 19076 raft_consensus.cc:482] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:13.505677 19076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:13.506523 19076 raft_consensus.cc:504] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02ec1636ea64456a0137253cbc03745" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } }
I20260620 04:34:13.506832 19076 leader_election.cc:302] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e02ec1636ea64456a0137253cbc03745; no voters:
I20260620 04:34:13.507115 19076 leader_election.cc:288] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:13.507210 19081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:13.507414 19081 raft_consensus.cc:686] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [term 1 LEADER]: Becoming Leader. State: Replica: e02ec1636ea64456a0137253cbc03745, State: Running, Role: LEADER
I20260620 04:34:13.507750 19081 consensus_queue.cc:237] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02ec1636ea64456a0137253cbc03745" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } }
I20260620 04:34:13.508211 19076 sys_catalog.cc:564] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:13.509305 19082 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e02ec1636ea64456a0137253cbc03745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02ec1636ea64456a0137253cbc03745" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } } }
I20260620 04:34:13.509509 19082 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:13.509306 19083 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e02ec1636ea64456a0137253cbc03745. Latest consensus state: current_term: 1 leader_uuid: "e02ec1636ea64456a0137253cbc03745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02ec1636ea64456a0137253cbc03745" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } } }
I20260620 04:34:13.509737 19083 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:13.509999 19090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:13.512821 19090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:13.518296 19090 catalog_manager.cc:1347] Generated new cluster ID: 6a6c11cfbbd24bb49d85af993a0ac937
I20260620 04:34:13.518368 19090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:13.550182 19090 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:13.551513 19090 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:13.556954 19090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745: Generated new TSK 0
I20260620 04:34:13.557654 19090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:13.568820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40699
--builtin_ntp_servers=127.0.98.20:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:34:13.679402 19100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:13.679675 19100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:13.679714 19100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:13.683715 19100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:13.683775 19100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:13.683835 19100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:34:13.688627 19100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:13.689698 19100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:13.691102 19100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:13.700378 19108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:13.700387 19106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:13.700398 19105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:13.700659 19100 server_base.cc:1034] running on GCE node
I20260620 04:34:13.701159 19100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:13.701800 19100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:13.702946 19100 hybrid_clock.cc:648] HybridClock initialized: now 1781930053702921 us; error 44 us; skew 500 ppm
I20260620 04:34:13.704737 19100 webserver.cc:460] Webserver started at http://127.0.98.1:40647/ using document root <none> and password file <none>
I20260620 04:34:13.705322 19100 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:13.705390 19100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:13.705586 19100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:13.707562 19100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "d37cbedd4a684fe58962943c3c4ce86f"
format_stamp: "Formatted at 2026-06-20 04:34:13 on dist-test-slave-rzqm"
server_key: "8040d43ca18d09372c3df6bd32371bb8"
server_key_iv: "eba2fdc81dd82536d26524fe8db38013"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.708097 19100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d37cbedd4a684fe58962943c3c4ce86f"
format_stamp: "Formatted at 2026-06-20 04:34:13 on dist-test-slave-rzqm"
server_key: "8040d43ca18d09372c3df6bd32371bb8"
server_key_iv: "eba2fdc81dd82536d26524fe8db38013"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.711777 19100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 04:34:13.714056 19114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:13.715308 19100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:13.715432 19100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "d37cbedd4a684fe58962943c3c4ce86f"
format_stamp: "Formatted at 2026-06-20 04:34:13 on dist-test-slave-rzqm"
server_key: "8040d43ca18d09372c3df6bd32371bb8"
server_key_iv: "eba2fdc81dd82536d26524fe8db38013"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.715523 19100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:13.760844 19100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:13.761618 19100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:13.761794 19100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:13.762408 19100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:13.763360 19100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:13.763417 19100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:13.763471 19100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:13.763496 19100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:13.772156 19100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39259
I20260620 04:34:13.772226 19227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39259 every 8 connection(s)
I20260620 04:34:13.773090 19100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:34:13.775084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 19100
I20260620 04:34:13.775207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:34:13.775475 392 external_mini_cluster.cc:1428] Setting key aa6afe168ba7231d0617dc97181d3192
I20260620 04:34:13.777691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40699
--builtin_ntp_servers=127.0.98.20:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:34:13.783073 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.774942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40103 (local address 127.0.98.62:40699)
0620 04:34:13.775100 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.775105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.776212 (+ 1107us) server_negotiation.cc:407] Connection header received
0620 04:34:13.777842 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.777846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.777899 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.777978 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:13.779095 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.779896 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.781059 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.781409 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.781522 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.782603 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:13.782607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:13.782610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:13.782624 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:13.782649 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:13.782660 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:13.782741 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.782851 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.782916 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:34:13.783867 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.775316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40699 (local address 127.0.98.1:40103)
0620 04:34:13.776059 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:13.776112 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:13.777614 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:13.778172 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:13.778190 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:13.778301 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:13.778950 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.778971 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.780029 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.780032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.780926 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.780935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.781578 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.781581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.781737 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.782350 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:34:13.782351 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:13.782392 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:34:13.782454 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:13.782737 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:13.782742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:13.782745 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 04:34:13.782831 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 04:34:13.783050 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":115,"threads_started":1}
I20260620 04:34:13.784984 19228 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40699
I20260620 04:34:13.785264 19228 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:13.785900 19228 heartbeater.cc:511] Master 127.0.98.62:40699 requested a full tablet report, sending...
I20260620 04:34:13.787423 19040 ts_manager.cc:194] Registered new tserver with Master: d37cbedd4a684fe58962943c3c4ce86f (127.0.98.1:39259)
I20260620 04:34:13.788573 19040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40103
W20260620 04:34:13.887728 19231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:13.888001 19231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:13.888039 19231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:13.892144 19231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:13.892213 19231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:13.892268 19231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:34:13.897065 19231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:13.898145 19231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:13.899530 19231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:13.908845 19237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:13.908874 19239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:13.908859 19236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:13.909226 19231 server_base.cc:1034] running on GCE node
I20260620 04:34:13.909641 19231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:13.910270 19231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:13.911469 19231 hybrid_clock.cc:648] HybridClock initialized: now 1781930053911452 us; error 27 us; skew 500 ppm
I20260620 04:34:13.913316 19231 webserver.cc:460] Webserver started at http://127.0.98.2:44093/ using document root <none> and password file <none>
I20260620 04:34:13.913862 19231 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:13.913928 19231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:13.914106 19231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:13.916091 19231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e"
format_stamp: "Formatted at 2026-06-20 04:34:13 on dist-test-slave-rzqm"
server_key: "37b5b85f41ea2474b04142ce5c301d06"
server_key_iv: "994758f3b2a68690d0d23e70ac0de367"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.916616 19231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e"
format_stamp: "Formatted at 2026-06-20 04:34:13 on dist-test-slave-rzqm"
server_key: "37b5b85f41ea2474b04142ce5c301d06"
server_key_iv: "994758f3b2a68690d0d23e70ac0de367"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.920323 19231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:34:13.922644 19245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:13.923837 19231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:13.923943 19231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e"
format_stamp: "Formatted at 2026-06-20 04:34:13 on dist-test-slave-rzqm"
server_key: "37b5b85f41ea2474b04142ce5c301d06"
server_key_iv: "994758f3b2a68690d0d23e70ac0de367"
server_key_version: "encryptionkey@0"
I20260620 04:34:13.924036 19231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:13.939370 19231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:13.940100 19231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:13.940274 19231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:13.940853 19231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:13.941810 19231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:13.941857 19231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:13.941910 19231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:13.941938 19231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:13.949901 19231 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35755
I20260620 04:34:13.949956 19358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35755 every 8 connection(s)
I20260620 04:34:13.950826 19231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:34:13.954100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 19231
I20260620 04:34:13.954229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:34:13.954564 392 external_mini_cluster.cc:1428] Setting key 1d9f92756bc00e5e9a6b68e4761a372c
I20260620 04:34:13.956605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40699
--builtin_ntp_servers=127.0.98.20:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:34:13.960384 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.952756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34605 (local address 127.0.98.62:40699)
0620 04:34:13.952907 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:13.952911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:13.954067 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 04:34:13.955690 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:13.955693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:13.955778 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:13.955864 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:13.956837 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.957399 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.958539 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.958768 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.958853 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.959939 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:13.959943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:13.959946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:13.959959 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:13.959978 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:13.959985 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:13.960042 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:13.960177 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 04:34:13.960236 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 04:34:13.961249 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:13.953190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40699 (local address 127.0.98.2:34605)
0620 04:34:13.953892 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:13.953950 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:13.955471 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:13.956077 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:13.956086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:13.956183 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:13.956681 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.956693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.957528 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.957530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.958423 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:13.958432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:13.958916 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:13.958921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:13.959074 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:13.959661 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:34:13.959663 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:13.959702 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:34:13.959784 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:13.960051 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:13.960060 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:13.960065 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 04:34:13.960172 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 04:34:13.960452 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":123,"threads_started":1}
I20260620 04:34:13.962307 19359 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40699
I20260620 04:34:13.962565 19359 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:13.963207 19359 heartbeater.cc:511] Master 127.0.98.62:40699 requested a full tablet report, sending...
I20260620 04:34:13.964306 19040 ts_manager.cc:194] Registered new tserver with Master: bb97a3f192fb40f8a65bcde74ccb0a3e (127.0.98.2:35755)
I20260620 04:34:13.964903 19040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34605
W20260620 04:34:14.068342 19362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:14.068603 19362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:14.068640 19362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:14.072664 19362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:14.072724 19362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:14.072788 19362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:34:14.077527 19362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:14.078552 19362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:14.080001 19362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:14.089109 19368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:14.089109 19370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:14.089103 19367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:14.089216 19362 server_base.cc:1034] running on GCE node
I20260620 04:34:14.089716 19362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:14.090327 19362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:14.091490 19362 hybrid_clock.cc:648] HybridClock initialized: now 1781930054091470 us; error 30 us; skew 500 ppm
I20260620 04:34:14.093334 19362 webserver.cc:460] Webserver started at http://127.0.98.3:39579/ using document root <none> and password file <none>
I20260620 04:34:14.093887 19362 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:14.093950 19362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:14.094139 19362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:14.096155 19362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "8929946309564a03b62e1afe0a5e9fd9"
format_stamp: "Formatted at 2026-06-20 04:34:14 on dist-test-slave-rzqm"
server_key: "164dade24d5e08ebee633decac96acee"
server_key_iv: "84d6a66efb7757c169b7ef3fc31b18d2"
server_key_version: "encryptionkey@0"
I20260620 04:34:14.096690 19362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8929946309564a03b62e1afe0a5e9fd9"
format_stamp: "Formatted at 2026-06-20 04:34:14 on dist-test-slave-rzqm"
server_key: "164dade24d5e08ebee633decac96acee"
server_key_iv: "84d6a66efb7757c169b7ef3fc31b18d2"
server_key_version: "encryptionkey@0"
I20260620 04:34:14.100436 19362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:34:14.102911 19376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:14.104213 19362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:34:14.104331 19362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "8929946309564a03b62e1afe0a5e9fd9"
format_stamp: "Formatted at 2026-06-20 04:34:14 on dist-test-slave-rzqm"
server_key: "164dade24d5e08ebee633decac96acee"
server_key_iv: "84d6a66efb7757c169b7ef3fc31b18d2"
server_key_version: "encryptionkey@0"
I20260620 04:34:14.104440 19362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:14.115346 19362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:14.115984 19362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:14.116147 19362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:14.116683 19362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:14.117635 19362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:14.117681 19362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:14.117731 19362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:14.117758 19362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:14.126189 19362 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45281
I20260620 04:34:14.126226 19489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45281 every 8 connection(s)
I20260620 04:34:14.127110 19362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:34:14.132632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 19362
I20260620 04:34:14.132747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:34:14.133057 392 external_mini_cluster.cc:1428] Setting key 3c6787c8677422c1c44917c686bc86c4
I20260620 04:34:14.141798 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.129104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44001 (local address 127.0.98.62:40699)
0620 04:34:14.129229 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:14.129233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:14.130319 (+ 1086us) server_negotiation.cc:407] Connection header received
0620 04:34:14.131668 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:14.131671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:14.131722 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:14.131830 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:14.132755 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.133543 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.134860 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.135135 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.135233 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.141228 (+ 5995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:14.141231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:14.141233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:14.141250 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:14.141274 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:14.141299 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:14.141363 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:14.141556 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 04:34:14.141638 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:34:14.142772 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.129492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40699 (local address 127.0.98.3:44001)
0620 04:34:14.130157 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:14.130199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:14.131456 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:14.131980 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:14.131987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:14.132077 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:14.132592 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.132609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.133681 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.133684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.134731 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.134742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.135275 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.135279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.135457 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.140777 (+ 5320us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:34:14.140818 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:14.140878 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:34:14.140970 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:14.141396 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:14.141409 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:14.141415 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:34:14.141541 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 04:34:14.141908 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":101,"threads_started":1}
I20260620 04:34:14.143975 19490 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40699
I20260620 04:34:14.144228 19490 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:14.144778 19490 heartbeater.cc:511] Master 127.0.98.62:40699 requested a full tablet report, sending...
I20260620 04:34:14.145929 19040 ts_manager.cc:194] Registered new tserver with Master: 8929946309564a03b62e1afe0a5e9fd9 (127.0.98.3:45281)
I20260620 04:34:14.146658 19040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44001
I20260620 04:34:14.146842 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:34:14.152413 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.148783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.0.98.62:40699)
0620 04:34:14.148930 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:14.148935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:14.149072 (+ 137us) server_negotiation.cc:407] Connection header received
0620 04:34:14.149175 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:14.149177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:14.149214 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:14.149271 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:14.149510 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.150059 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.150943 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.151153 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.151247 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.152027 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:14.152030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:14.152032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:14.152044 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:14.152061 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:14.152069 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:14.152161 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:14.152237 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 04:34:14.152270 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 04:34:14.155241 19040 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:34:14.158499 19040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:34:14.172621 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.166716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39259 (local address 127.0.0.1:53746)
0620 04:34:14.167472 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:14.167485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:14.167572 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:14.168057 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:14.168060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:14.168075 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:14.168146 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.168152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.169756 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.169759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.170955 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.170964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.172021 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.172024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.172136 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.172204 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 04:34:14.172325 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 04:34:14.172458 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":148,"thread_start_us":81,"threads_started":1}
I20260620 04:34:14.172631 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.166886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.0.98.1:39259)
0620 04:34:14.167630 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:14.167641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:14.167661 (+ 20us) server_negotiation.cc:407] Connection header received
0620 04:34:14.167729 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:14.167733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:14.167917 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:14.168266 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:14.168389 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.169580 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.171092 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.171898 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.172007 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.172214 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:14.172366 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 04:34:14.172464 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":87,"threads_started":1}
I20260620 04:34:14.172789 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.166650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.0.98.2:35755)
0620 04:34:14.167268 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:14.167279 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:14.167305 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:34:14.167502 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:14.167508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:14.167724 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:14.168220 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:14.168372 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.169580 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.171286 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.172009 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.172118 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.172273 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:14.172536 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 04:34:14.172636 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":128,"threads_started":1}
I20260620 04:34:14.173197 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.166503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35755 (local address 127.0.0.1:37512)
0620 04:34:14.167168 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:14.167221 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:14.167346 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:14.167894 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:14.167900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:14.167922 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:14.168069 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.168076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.169723 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.169726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.171162 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.171168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.172116 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.172119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.172192 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.172211 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:34:14.172318 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 04:34:14.172459 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":91,"thread_start_us":70,"threads_started":1}
I20260620 04:34:14.173667 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.167364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.0.98.3:45281)
0620 04:34:14.167793 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:14.167805 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:14.168135 (+ 330us) server_negotiation.cc:407] Connection header received
0620 04:34:14.168215 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:14.168220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:14.168423 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:14.168557 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:14.168766 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.169560 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.172342 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.173124 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.173207 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.173377 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:14.173430 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 04:34:14.173517 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":83,"threads_started":1}
I20260620 04:34:14.173596 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.167203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45281 (local address 127.0.0.1:43776)
0620 04:34:14.167951 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:14.167963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:14.168044 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:14.168550 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:14.168553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:14.168570 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:14.168662 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.168668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.169662 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.169666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.171156 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.171167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.173235 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.173238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.173322 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.173344 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:34:14.173407 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:34:14.173478 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":104,"thread_start_us":84,"threads_started":1}
I20260620 04:34:14.176517 19162 tablet_service.cc:1467] Processing CreateTablet for tablet c445815384124694ad03c0faee2a6482 (DEFAULT_TABLE table=test-table [id=684627d8eaba4b75a0dcbdb0a58a7092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:14.176764 19293 tablet_service.cc:1467] Processing CreateTablet for tablet c445815384124694ad03c0faee2a6482 (DEFAULT_TABLE table=test-table [id=684627d8eaba4b75a0dcbdb0a58a7092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:14.177749 19162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c445815384124694ad03c0faee2a6482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:14.177749 19293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c445815384124694ad03c0faee2a6482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:14.179433 19424 tablet_service.cc:1467] Processing CreateTablet for tablet c445815384124694ad03c0faee2a6482 (DEFAULT_TABLE table=test-table [id=684627d8eaba4b75a0dcbdb0a58a7092]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:14.180263 19424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c445815384124694ad03c0faee2a6482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:14.185390 19508 tablet_bootstrap.cc:492] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f: Bootstrap starting.
I20260620 04:34:14.185835 19507 tablet_bootstrap.cc:492] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: Bootstrap starting.
I20260620 04:34:14.187265 19509 tablet_bootstrap.cc:492] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: Bootstrap starting.
I20260620 04:34:14.188172 19507 tablet_bootstrap.cc:654] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:14.189116 19509 tablet_bootstrap.cc:654] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:14.189079 19507 log.cc:824] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:14.189920 19509 log.cc:824] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:14.191329 19507 tablet_bootstrap.cc:492] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: No bootstrap required, opened a new log
I20260620 04:34:14.191494 19508 tablet_bootstrap.cc:654] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:14.191532 19507 ts_tablet_manager.cc:1397] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 04:34:14.191591 19509 tablet_bootstrap.cc:492] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: No bootstrap required, opened a new log
I20260620 04:34:14.191762 19509 ts_tablet_manager.cc:1397] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 04:34:14.192286 19508 log.cc:824] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:14.194357 19508 tablet_bootstrap.cc:492] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f: No bootstrap required, opened a new log
I20260620 04:34:14.194552 19508 ts_tablet_manager.cc:1397] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 04:34:14.195127 19509 raft_consensus.cc:348] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35755 } } peers { permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 } } peers { permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } }
I20260620 04:34:14.195431 19509 raft_consensus.cc:374] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:14.195501 19509 raft_consensus.cc:729] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8929946309564a03b62e1afe0a5e9fd9, State: Initialized, Role: FOLLOWER
I20260620 04:34:14.195645 19507 raft_consensus.cc:348] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35755 } } peers { permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 } } peers { permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } }
I20260620 04:34:14.195911 19507 raft_consensus.cc:374] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:14.195959 19507 raft_consensus.cc:729] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb97a3f192fb40f8a65bcde74ccb0a3e, State: Initialized, Role: FOLLOWER
I20260620 04:34:14.195999 19509 consensus_queue.cc:260] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35755 } } peers { permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 } } peers { permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } }
I20260620 04:34:14.196420 19507 consensus_queue.cc:260] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35755 } } peers { permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 } } peers { permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } }
I20260620 04:34:14.197082 19509 ts_tablet_manager.cc:1428] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 04:34:14.197413 19490 heartbeater.cc:503] Master 127.0.98.62:40699 was elected leader, sending a full tablet report...
I20260620 04:34:14.197415 19359 heartbeater.cc:503] Master 127.0.98.62:40699 was elected leader, sending a full tablet report...
I20260620 04:34:14.198753 19507 ts_tablet_manager.cc:1428] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 04:34:14.198765 19508 raft_consensus.cc:348] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35755 } } peers { permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 } } peers { permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } }
I20260620 04:34:14.199079 19508 raft_consensus.cc:374] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:14.199142 19508 raft_consensus.cc:729] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37cbedd4a684fe58962943c3c4ce86f, State: Initialized, Role: FOLLOWER
I20260620 04:34:14.199538 19508 consensus_queue.cc:260] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35755 } } peers { permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 } } peers { permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } }
I20260620 04:34:14.200122 19228 heartbeater.cc:503] Master 127.0.98.62:40699 was elected leader, sending a full tablet report...
I20260620 04:34:14.200428 19508 ts_tablet_manager.cc:1428] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260620 04:34:14.201745 19360 tablet.cc:2369] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 04:34:14.274255 19229 tablet.cc:2369] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:34:14.361225 19514 raft_consensus.cc:482] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:34:14.361424 19514 raft_consensus.cc:504] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35755 } } peers { permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 } } peers { permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } }
I20260620 04:34:14.362498 19514 leader_election.cc:288] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d37cbedd4a684fe58962943c3c4ce86f (127.0.98.1:39259), 8929946309564a03b62e1afe0a5e9fd9 (127.0.98.3:45281)
I20260620 04:34:14.366225 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.362740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39259 (local address 127.0.98.2:53593)
0620 04:34:14.362857 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:14.362873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:14.362966 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:14.363223 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:14.363226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:14.363245 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:14.363349 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.363355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.364122 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.364125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.365176 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.365189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.365920 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.365922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.365994 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.366014 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:34:14.366064 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 04:34:14.366118 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 04:34:14.366294 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.362830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53593 (local address 127.0.98.1:39259)
0620 04:34:14.362943 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:14.362947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:14.362963 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:34:14.363068 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:14.363071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:14.363119 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:14.363195 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:14.363461 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.364019 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.365335 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.365798 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.365914 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.365976 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:14.366096 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 04:34:14.366163 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:34:14.366938 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.363147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36777 (local address 127.0.98.3:45281)
0620 04:34:14.363265 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:14.363269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:14.363495 (+ 226us) server_negotiation.cc:407] Connection header received
0620 04:34:14.363590 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:14.363593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:14.363633 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:14.363695 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:14.363997 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.364508 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.365803 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.366410 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.366509 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.366563 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:14.366784 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 04:34:14.366850 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 04:34:14.366951 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.363015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45281 (local address 127.0.98.2:36777)
0620 04:34:14.363409 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:14.363420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:14.363507 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:14.363745 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:14.363748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:14.363765 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:14.363897 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.363903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.364617 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.364620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.365681 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:14.365690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.366593 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.366596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:14.366663 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.366683 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 04:34:14.366754 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:34:14.366814 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":111,"threads_started":1}
I20260620 04:34:14.366993 19182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c445815384124694ad03c0faee2a6482" candidate_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37cbedd4a684fe58962943c3c4ce86f" is_pre_election: true
I20260620 04:34:14.367298 19182 raft_consensus.cc:2454] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb97a3f192fb40f8a65bcde74ccb0a3e in term 0.
I20260620 04:34:14.367674 19444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c445815384124694ad03c0faee2a6482" candidate_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8929946309564a03b62e1afe0a5e9fd9" is_pre_election: true
I20260620 04:34:14.367790 19249 leader_election.cc:302] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb97a3f192fb40f8a65bcde74ccb0a3e, d37cbedd4a684fe58962943c3c4ce86f; no voters:
I20260620 04:34:14.367976 19444 raft_consensus.cc:2454] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb97a3f192fb40f8a65bcde74ccb0a3e in term 0.
I20260620 04:34:14.368094 19514 raft_consensus.cc:2781] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:34:14.368183 19514 raft_consensus.cc:482] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:34:14.368230 19514 raft_consensus.cc:3037] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:14.369318 19514 raft_consensus.cc:504] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35755 } } peers { permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 } } peers { permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } }
I20260620 04:34:14.369634 19514 leader_election.cc:288] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [CANDIDATE]: Term 1 election: Requested vote from peers d37cbedd4a684fe58962943c3c4ce86f (127.0.98.1:39259), 8929946309564a03b62e1afe0a5e9fd9 (127.0.98.3:45281)
I20260620 04:34:14.369956 19182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c445815384124694ad03c0faee2a6482" candidate_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37cbedd4a684fe58962943c3c4ce86f"
I20260620 04:34:14.370020 19444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c445815384124694ad03c0faee2a6482" candidate_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8929946309564a03b62e1afe0a5e9fd9"
I20260620 04:34:14.370074 19182 raft_consensus.cc:3037] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:14.370136 19444 raft_consensus.cc:3037] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:14.371105 19182 raft_consensus.cc:2454] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb97a3f192fb40f8a65bcde74ccb0a3e in term 1.
I20260620 04:34:14.371167 19444 raft_consensus.cc:2454] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb97a3f192fb40f8a65bcde74ccb0a3e in term 1.
I20260620 04:34:14.371381 19249 leader_election.cc:302] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb97a3f192fb40f8a65bcde74ccb0a3e, d37cbedd4a684fe58962943c3c4ce86f; no voters:
I20260620 04:34:14.371559 19514 raft_consensus.cc:2781] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:14.371783 19514 raft_consensus.cc:686] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 1 LEADER]: Becoming Leader. State: Replica: bb97a3f192fb40f8a65bcde74ccb0a3e, State: Running, Role: LEADER
I20260620 04:34:14.372092 19514 consensus_queue.cc:237] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35755 } } peers { permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 } } peers { permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } }
I20260620 04:34:14.375324 19040 catalog_manager.cc:5495] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e reported cstate change: term changed from 0 to 1, leader changed from <none> to bb97a3f192fb40f8a65bcde74ccb0a3e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb97a3f192fb40f8a65bcde74ccb0a3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } health_report { overall_health: UNKNOWN } } }
W20260620 04:34:14.378031 19491 tablet.cc:2369] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:34:14.410741 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.406471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.0.98.2:35755)
0620 04:34:14.406625 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:14.406630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:14.406642 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:34:14.406711 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:14.406713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:14.406767 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:14.406862 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:14.407136 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.407933 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.409050 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:14.409411 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:14.409515 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:14.409921 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:14.410385 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:14.410497 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:14.410584 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 04:34:14.410617 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 04:34:14.416512 19444 raft_consensus.cc:1261] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9 [term 1 FOLLOWER]: Refusing update from remote peer bb97a3f192fb40f8a65bcde74ccb0a3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:14.416512 19182 raft_consensus.cc:1261] T c445815384124694ad03c0faee2a6482 P d37cbedd4a684fe58962943c3c4ce86f [term 1 FOLLOWER]: Refusing update from remote peer bb97a3f192fb40f8a65bcde74ccb0a3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:14.417236 19517 consensus_queue.cc:1035] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37cbedd4a684fe58962943c3c4ce86f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:14.417412 19514 consensus_queue.cc:1035] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8929946309564a03b62e1afe0a5e9fd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:14.425613 19521 mvcc.cc:205] Tried to move back new op lower bound from 7298785502882582528 to 7298785502710427648. Current Snapshot: MvccSnapshot[applied={T|T < 7298785502882582528}]
I20260620 04:34:14.427491 19522 mvcc.cc:205] Tried to move back new op lower bound from 7298785502882582528 to 7298785502710427648. Current Snapshot: MvccSnapshot[applied={T|T < 7298785502882582528}]
I20260620 04:34:14.433770 19038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:14.433949 19038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:14.436290 19038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e02ec1636ea64456a0137253cbc03745: Sending DeleteTablet for 3 replicas of tablet c445815384124694ad03c0faee2a6482
I20260620 04:34:14.437160 19424 tablet_service.cc:1514] Processing DeleteTablet for tablet c445815384124694ad03c0faee2a6482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:14 UTC) from {username='slave'} at 127.0.0.1:43776
I20260620 04:34:14.437197 19293 tablet_service.cc:1514] Processing DeleteTablet for tablet c445815384124694ad03c0faee2a6482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:14 UTC) from {username='slave'} at 127.0.0.1:37512
I20260620 04:34:14.437300 19162 tablet_service.cc:1514] Processing DeleteTablet for tablet c445815384124694ad03c0faee2a6482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:14 UTC) from {username='slave'} at 127.0.0.1:53746
I20260620 04:34:14.437687 19531 tablet_replica.cc:330] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: stopping tablet replica
I20260620 04:34:14.438071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 19100
I20260620 04:34:14.440151 19531 raft_consensus.cc:2229] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:34:14.440515 19531 raft_consensus.cc:2258] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:14.441025 19530 tablet_replica.cc:330] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: stopping tablet replica
I20260620 04:34:14.441322 19530 raft_consensus.cc:2229] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:34:14.441623 19530 raft_consensus.cc:2258] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:14.443147 19530 ts_tablet_manager.cc:1905] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 04:34:14.445201 19027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39259 (error 108)
I20260620 04:34:14.445933 19530 ts_tablet_manager.cc:1918] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:14.446110 19530 log.cc:1196] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/c445815384124694ad03c0faee2a6482
I20260620 04:34:14.446480 19530 ts_tablet_manager.cc:1939] T c445815384124694ad03c0faee2a6482 P 8929946309564a03b62e1afe0a5e9fd9: Deleting consensus metadata
I20260620 04:34:14.447279 19531 ts_tablet_manager.cc:1905] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:14.447743 19027 catalog_manager.cc:4841] TS 8929946309564a03b62e1afe0a5e9fd9 (127.0.98.3:45281): tablet c445815384124694ad03c0faee2a6482 (table test-table [id=684627d8eaba4b75a0dcbdb0a58a7092]) successfully deleted
I20260620 04:34:14.449427 19531 ts_tablet_manager.cc:1918] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:14.449506 19531 log.cc:1196] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/c445815384124694ad03c0faee2a6482
I20260620 04:34:14.449783 19531 ts_tablet_manager.cc:1939] T c445815384124694ad03c0faee2a6482 P bb97a3f192fb40f8a65bcde74ccb0a3e: Deleting consensus metadata
W20260620 04:34:14.449797 19501 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:14.445930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39259 (local address 127.0.0.1:53760)
0620 04:34:14.446058 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:14.446074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:14.446161 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:14.449687 (+ 3526us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39259: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 04:34:14.450004 19027 catalog_manager.cc:4584] TS d37cbedd4a684fe58962943c3c4ce86f (127.0.98.1:39259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c445815384124694ad03c0faee2a6482: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39259: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 04:34:14.450263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 19231
I20260620 04:34:14.455993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 19362
I20260620 04:34:14.457326 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:14.456945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35755 (local address 127.0.0.1:37530)
0620 04:34:14.457151 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:14.457233 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 04:34:14.460861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 19009
2026-06-20T04:34:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1171 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:14 dist-test-slave-rzqm krb5kdc[19536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:14 dist-test-slave-rzqm krb5kdc[19536](info): set up 2 sockets
Jun 20 04:34:14 dist-test-slave-rzqm krb5kdc[19536](info): commencing operation
krb5kdc: starting...
W20260620 04:34:15.416790 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:15 dist-test-slave-rzqm krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:15.528235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35125
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:15.635514 19549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:15.635799 19549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:15.635866 19549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:15.640019 19549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:15.640089 19549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:15.640106 19549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:15.640121 19549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:15.640136 19549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:15.645473 19549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:15.646530 19549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:15.647571 19549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:34:15.656057 19549 server_base.cc:1034] running on GCE node
W20260620 04:34:15.656010 19554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:15.656010 19555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:15.656008 19557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:15.656924 19549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:15.658104 19549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:15.659296 19549 hybrid_clock.cc:648] HybridClock initialized: now 1781930055659264 us; error 43 us; skew 500 ppm
Jun 20 04:34:15 dist-test-slave-rzqm krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:15.662103 19549 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:15.663038 19549 webserver.cc:460] Webserver started at http://127.0.98.62:35923/ using document root <none> and password file <none>
I20260620 04:34:15.663601 19549 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:15.663669 19549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:15.663885 19549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:15.666018 19549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "f0079689aad24596a88033552fc51688"
format_stamp: "Formatted at 2026-06-20 04:34:15 on dist-test-slave-rzqm"
server_key: "de44b5237e04a353299beecf5df0116c"
server_key_iv: "554c6c2140bce86349b44fe032533370"
server_key_version: "encryptionkey@0"
I20260620 04:34:15.666579 19549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0079689aad24596a88033552fc51688"
format_stamp: "Formatted at 2026-06-20 04:34:15 on dist-test-slave-rzqm"
server_key: "de44b5237e04a353299beecf5df0116c"
server_key_iv: "554c6c2140bce86349b44fe032533370"
server_key_version: "encryptionkey@0"
I20260620 04:34:15.670428 19549 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:34:15.672955 19564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:15.674245 19549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:15.674397 19549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "f0079689aad24596a88033552fc51688"
format_stamp: "Formatted at 2026-06-20 04:34:15 on dist-test-slave-rzqm"
server_key: "de44b5237e04a353299beecf5df0116c"
server_key_iv: "554c6c2140bce86349b44fe032533370"
server_key_version: "encryptionkey@0"
I20260620 04:34:15.674504 19549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:15.685222 19549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:15.688319 19549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:15.688503 19549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:15.696724 19549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35125
I20260620 04:34:15.696751 19616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35125 every 8 connection(s)
I20260620 04:34:15.698207 19549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:15.701278 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:15.704030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 19549
I20260620 04:34:15.704187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:15.704452 392 external_mini_cluster.cc:1428] Setting key f46e9f09542e897903b1c4e577da3b46
I20260620 04:34:15.707054 19617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688: Bootstrap starting.
I20260620 04:34:15.709546 19617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:15.710271 19617 log.cc:824] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:15.712126 19617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688: No bootstrap required, opened a new log
Jun 20 04:34:15 dist-test-slave-rzqm krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:15.715346 19617 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0079689aad24596a88033552fc51688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260620 04:34:15.715565 19617 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:15.715600 19617 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0079689aad24596a88033552fc51688, State: Initialized, Role: FOLLOWER
I20260620 04:34:15.716058 19617 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0079689aad24596a88033552fc51688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260620 04:34:15.716224 19617 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:15.716289 19617 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:15.716391 19617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:15.717401 19617 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0079689aad24596a88033552fc51688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260620 04:34:15.717765 19617 leader_election.cc:302] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0079689aad24596a88033552fc51688; no voters:
I20260620 04:34:15.718113 19617 leader_election.cc:288] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:15.718207 19622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:15.718372 19622 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [term 1 LEADER]: Becoming Leader. State: Replica: f0079689aad24596a88033552fc51688, State: Running, Role: LEADER
I20260620 04:34:15.718715 19622 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0079689aad24596a88033552fc51688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260620 04:34:15.719018 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:15.705793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.0.98.62:35125)
0620 04:34:15.706217 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:15.706233 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:15.706268 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:34:15.706887 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:15.706904 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:15.707258 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:15.707550 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:15.708453 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:15.709538 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:15.710385 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:15.711008 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:15.711159 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:15.713716 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:15.713722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:15.713734 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:15.713761 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:15.715909 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:15.716373 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:15.716379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:15.716385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:15.716527 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:15.716796 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:15.716798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:15.716799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:15.717234 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:15.717390 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:15.717694 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 04:34:15.718018 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20260620 04:34:15.719386 19617 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:15.720331 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0079689aad24596a88033552fc51688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0079689aad24596a88033552fc51688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } } }
I20260620 04:34:15.720472 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:15.720695 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0079689aad24596a88033552fc51688. Latest consensus state: current_term: 1 leader_uuid: "f0079689aad24596a88033552fc51688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0079689aad24596a88033552fc51688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } } }
I20260620 04:34:15.720781 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:15.721675 19631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:15.724321 19631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:15.730029 19631 catalog_manager.cc:1347] Generated new cluster ID: 21663d02d9064403a0b5f9a83bbe3a65
I20260620 04:34:15.730108 19631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:15.742622 19631 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:15.743908 19631 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:15.754463 19631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688: Generated new TSK 0
I20260620 04:34:15.755309 19631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 04:34:15.791064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35125
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:15.899614 19643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:15.899885 19643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:15.899938 19643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:15.903919 19643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:15.903990 19643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:15.904049 19643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:34:15.909366 19643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:15.910449 19643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:15.911476 19643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:15.920770 19651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:15.920753 19649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:15.920770 19648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:15.921057 19643 server_base.cc:1034] running on GCE node
I20260620 04:34:15.921475 19643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:15.922164 19643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:15.923318 19643 hybrid_clock.cc:648] HybridClock initialized: now 1781930055923293 us; error 47 us; skew 500 ppm
Jun 20 04:34:15 dist-test-slave-rzqm krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:15.925997 19643 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 04:34:15.926880 19643 webserver.cc:460] Webserver started at http://127.0.98.1:38117/ using document root <none> and password file <none>
I20260620 04:34:15.927390 19643 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:15.927459 19643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:15.927616 19643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:15.929623 19643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "4837953e856a40e090547aaae334e263"
format_stamp: "Formatted at 2026-06-20 04:34:15 on dist-test-slave-rzqm"
server_key: "002d205815ac76d273bb62da09fbe21c"
server_key_iv: "33dea5caa5dd5824f429413f22ea3c43"
server_key_version: "encryptionkey@0"
I20260620 04:34:15.930127 19643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4837953e856a40e090547aaae334e263"
format_stamp: "Formatted at 2026-06-20 04:34:15 on dist-test-slave-rzqm"
server_key: "002d205815ac76d273bb62da09fbe21c"
server_key_iv: "33dea5caa5dd5824f429413f22ea3c43"
server_key_version: "encryptionkey@0"
I20260620 04:34:15.933900 19643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:34:15.936254 19658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:15.937557 19643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:15.937654 19643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "4837953e856a40e090547aaae334e263"
format_stamp: "Formatted at 2026-06-20 04:34:15 on dist-test-slave-rzqm"
server_key: "002d205815ac76d273bb62da09fbe21c"
server_key_iv: "33dea5caa5dd5824f429413f22ea3c43"
server_key_version: "encryptionkey@0"
I20260620 04:34:15.937739 19643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:15.954984 19643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:15.958016 19643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:15.958174 19643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:15.958734 19643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:15.959583 19643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:15.959642 19643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:15.959689 19643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:15.959715 19643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:15.968338 19643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44341
I20260620 04:34:15.968370 19771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44341 every 8 connection(s)
I20260620 04:34:15.969256 19643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:34:15.977241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 19643
I20260620 04:34:15.977352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:34:15.977593 392 external_mini_cluster.cc:1428] Setting key 2a070a723f865cf8599148f023d1c836
Jun 20 04:34:15 dist-test-slave-rzqm krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:15.982975 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:15.971078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36963 (local address 127.0.98.62:35125)
0620 04:34:15.971221 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:15.971226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:15.972280 (+ 1054us) server_negotiation.cc:407] Connection header received
0620 04:34:15.973158 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:15.973161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:15.973208 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:15.973270 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:15.974452 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:15.974995 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:15.976161 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:15.976381 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:15.976458 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:15.979175 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:15.979179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:15.979182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:15.979208 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:15.980790 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:15.981365 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:15.981368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:15.981370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:15.981477 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:15.981856 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:15.981859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:15.981861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:15.982090 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:15.982182 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:15.982689 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 04:34:15.982814 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:34:15.983865 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:15.971459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35125 (local address 127.0.98.1:36963)
0620 04:34:15.972136 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:15.972178 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:15.972942 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:15.973424 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:15.973431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:15.973831 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:15.974320 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:15.974334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:15.975118 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:15.975121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:15.976066 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:15.976076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:15.976522 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:15.976525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:15.976675 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:15.977296 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:15.977319 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:15.979007 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:15.980938 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:15.980945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:15.980962 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:15.981262 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:15.981597 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:15.981600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:15.981601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:15.981754 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:15.982183 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:15.982188 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:15.982453 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 04:34:15.982609 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 04:34:15.982878 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":113,"threads_started":1}
I20260620 04:34:15.985121 19772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35125
I20260620 04:34:15.985407 19772 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:15.986007 19772 heartbeater.cc:511] Master 127.0.98.62:35125 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 04:34:15.987702 19578 ts_manager.cc:194] Registered new tserver with Master: 4837953e856a40e090547aaae334e263 (127.0.98.1:44341)
I20260620 04:34:15.989042 19578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 04:34:16.003334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35125
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:16.112911 19777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:16.113255 19777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:16.113319 19777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:16.117364 19777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:16.117426 19777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:16.117506 19777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:34:16.123023 19777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:16.124079 19777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:16.125268 19777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:16.134795 19785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:16.134795 19782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:16.134833 19777 server_base.cc:1034] running on GCE node
W20260620 04:34:16.134872 19783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:16.135443 19777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:16.136076 19777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:16.137239 19777 hybrid_clock.cc:648] HybridClock initialized: now 1781930056137209 us; error 42 us; skew 500 ppm
Jun 20 04:34:16 dist-test-slave-rzqm krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:16.139959 19777 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 04:34:16.140919 19777 webserver.cc:460] Webserver started at http://127.0.98.2:33949/ using document root <none> and password file <none>
I20260620 04:34:16.141523 19777 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:16.141597 19777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:16.141793 19777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:16.143852 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae5396e2b76441808ceec06b043a4b3a"
format_stamp: "Formatted at 2026-06-20 04:34:16 on dist-test-slave-rzqm"
server_key: "3d7216cea6faa8e54472dc7185d3e2e6"
server_key_iv: "2c96168961093d86f031eb92bdc5e130"
server_key_version: "encryptionkey@0"
I20260620 04:34:16.144380 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae5396e2b76441808ceec06b043a4b3a"
format_stamp: "Formatted at 2026-06-20 04:34:16 on dist-test-slave-rzqm"
server_key: "3d7216cea6faa8e54472dc7185d3e2e6"
server_key_iv: "2c96168961093d86f031eb92bdc5e130"
server_key_version: "encryptionkey@0"
I20260620 04:34:16.147990 19777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:34:16.150619 19792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:16.151923 19777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:16.152034 19777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "ae5396e2b76441808ceec06b043a4b3a"
format_stamp: "Formatted at 2026-06-20 04:34:16 on dist-test-slave-rzqm"
server_key: "3d7216cea6faa8e54472dc7185d3e2e6"
server_key_iv: "2c96168961093d86f031eb92bdc5e130"
server_key_version: "encryptionkey@0"
I20260620 04:34:16.152130 19777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:16.171903 19777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:16.174799 19777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:16.174975 19777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:16.175542 19777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:16.176348 19777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:16.176397 19777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:16.176458 19777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:16.176482 19777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:16.184973 19777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43457
I20260620 04:34:16.184994 19905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43457 every 8 connection(s)
I20260620 04:34:16.185868 19777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:34:16.189651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 19777
I20260620 04:34:16.189772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:34:16.190035 392 external_mini_cluster.cc:1428] Setting key 17583ce48cd082cf6e58f65baff9c8cc
Jun 20 04:34:16 dist-test-slave-rzqm krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 04:34:16.199101 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.187630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45315 (local address 127.0.98.62:35125)
0620 04:34:16.187767 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:16.187770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:16.188637 (+ 867us) server_negotiation.cc:407] Connection header received
0620 04:34:16.189582 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:16.189585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:16.189644 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:16.189715 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:16.191119 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.191725 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.192885 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.193151 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.193235 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.195615 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:16.195619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:16.195622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:16.195647 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:16.197091 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:16.197568 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:16.197571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:16.197573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:16.197654 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:16.197972 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:16.197976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:16.197978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:16.198169 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:16.198257 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:16.198791 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 04:34:16.198931 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:34:16.200075 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.188001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35125 (local address 127.0.98.2:45315)
0620 04:34:16.188478 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:16.188530 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:16.189377 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:16.189901 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:16.189908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:16.190291 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:16.190979 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.190998 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.191857 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.191860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.192755 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.192766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.193298 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.193302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.193472 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.194044 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:16.194061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:16.195448 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:16.197194 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:16.197200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:16.197217 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:16.197473 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:16.197736 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:16.197738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:16.197740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:16.197886 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:16.198267 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:16.198273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:16.198550 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 04:34:16.198726 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 04:34:16.199083 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 04:34:16.201367 19906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35125
I20260620 04:34:16.201609 19906 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:16.202139 19906 heartbeater.cc:511] Master 127.0.98.62:35125 requested a full tablet report, sending...
I20260620 04:34:16.203274 19578 ts_manager.cc:194] Registered new tserver with Master: ae5396e2b76441808ceec06b043a4b3a (127.0.98.2:43457)
I20260620 04:34:16.203864 19578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 04:34:16.213677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35125
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:16.323601 19911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:16.323873 19911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:16.323921 19911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:16.328223 19911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:16.328295 19911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:16.328363 19911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:34:16.333887 19911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:16.334934 19911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:16.335958 19911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:16.345649 19919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:16.345674 19917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:16.345654 19916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:16.346042 19911 server_base.cc:1034] running on GCE node
I20260620 04:34:16.346405 19911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:16.347021 19911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:16.348170 19911 hybrid_clock.cc:648] HybridClock initialized: now 1781930056348142 us; error 42 us; skew 500 ppm
Jun 20 04:34:16 dist-test-slave-rzqm krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:16.350858 19911 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 04:34:16.351658 19911 webserver.cc:460] Webserver started at http://127.0.98.3:38315/ using document root <none> and password file <none>
I20260620 04:34:16.352176 19911 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:16.352233 19911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:16.352391 19911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:16.354377 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "87c3b5da17ee4923901971134abf1ee5"
format_stamp: "Formatted at 2026-06-20 04:34:16 on dist-test-slave-rzqm"
server_key: "49fec102849b8ea827b0badb7e9932d7"
server_key_iv: "f9b7bf3aa324137e31ee2605a13c6a65"
server_key_version: "encryptionkey@0"
I20260620 04:34:16.354903 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87c3b5da17ee4923901971134abf1ee5"
format_stamp: "Formatted at 2026-06-20 04:34:16 on dist-test-slave-rzqm"
server_key: "49fec102849b8ea827b0badb7e9932d7"
server_key_iv: "f9b7bf3aa324137e31ee2605a13c6a65"
server_key_version: "encryptionkey@0"
I20260620 04:34:16.358458 19911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:34:16.360836 19926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:16.362151 19911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:16.362257 19911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "87c3b5da17ee4923901971134abf1ee5"
format_stamp: "Formatted at 2026-06-20 04:34:16 on dist-test-slave-rzqm"
server_key: "49fec102849b8ea827b0badb7e9932d7"
server_key_iv: "f9b7bf3aa324137e31ee2605a13c6a65"
server_key_version: "encryptionkey@0"
I20260620 04:34:16.362353 19911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:16.402856 19911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:16.405618 19911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:16.405793 19911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:16.406368 19911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:16.407192 19911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:16.407239 19911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:16.407291 19911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:16.407317 19911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:16.416143 19911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38921
I20260620 04:34:16.416210 20039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38921 every 8 connection(s)
I20260620 04:34:16.417071 19911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:34:16.420414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 19911
I20260620 04:34:16.420542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:34:16.420851 392 external_mini_cluster.cc:1428] Setting key 63d4eb28aeb1a4820d9a90f154b318fd
Jun 20 04:34:16 dist-test-slave-rzqm krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:16.430387 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.418894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49811 (local address 127.0.98.62:35125)
0620 04:34:16.420655 (+ 1761us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:16.420660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:16.420696 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:34:16.420925 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:16.420928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:16.420993 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:16.421110 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:16.422595 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.423281 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.424417 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.424639 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.424720 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.427215 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:16.427218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:16.427220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:16.427244 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:16.428530 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:16.429045 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:16.429048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:16.429050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:16.429133 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:16.429439 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:16.429441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:16.429443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:16.429632 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:16.429705 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:16.430152 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 04:34:16.430263 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663}
I20260620 04:34:16.431247 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.419288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35125 (local address 127.0.98.3:49811)
0620 04:34:16.419747 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:16.419791 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:16.420722 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:16.421277 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:16.421286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:16.421792 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:16.422441 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.422463 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.423399 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.423408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.424296 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.424305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.424755 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.424758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.424905 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.425621 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:16.425638 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:16.427059 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:16.428641 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:16.428646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:16.428658 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:16.428905 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:16.429215 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:16.429217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:16.429218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:16.429353 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:16.429706 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:16.429710 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:16.429947 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 04:34:16.430101 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 04:34:16.430362 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
I20260620 04:34:16.432396 20040 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35125
I20260620 04:34:16.432634 20040 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:16.433123 20040 heartbeater.cc:511] Master 127.0.98.62:35125 requested a full tablet report, sending...
I20260620 04:34:16.434204 19578 ts_manager.cc:194] Registered new tserver with Master: 87c3b5da17ee4923901971134abf1ee5 (127.0.98.3:38921)
I20260620 04:34:16.434888 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:34:16.434947 19578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49811
I20260620 04:34:16.443423 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.436867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.0.98.62:35125)
0620 04:34:16.436974 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:16.436977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:16.437165 (+ 188us) server_negotiation.cc:407] Connection header received
0620 04:34:16.437254 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:16.437256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:16.437292 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:16.437353 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:16.438022 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.438513 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.439319 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.439532 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.439609 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.440739 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:16.440741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:16.440743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:16.440770 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:16.442142 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:16.442494 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:16.442496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:16.442498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:16.442564 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:16.442755 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:16.442757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:16.442758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:16.442975 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:16.443078 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:16.443219 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 04:34:16.443308 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:34:16.446166 19578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:34:16.449147 19578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:34:16.463575 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.457669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44341 (local address 127.0.0.1:44904)
0620 04:34:16.458432 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:16.458444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:16.458525 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:16.459109 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:16.459115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:16.459156 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:16.459495 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.459501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.460814 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.460817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.462084 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.462093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.463037 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.463041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.463146 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.463192 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 04:34:16.463297 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 04:34:16.463416 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":202,"spinlock_wait_cycles":7296,"thread_start_us":105,"threads_started":1}
I20260620 04:34:16.463600 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.458103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.0.98.1:44341)
0620 04:34:16.458620 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:16.458634 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:16.458656 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:34:16.458729 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:16.458734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:16.458952 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:16.459112 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:16.459605 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.460699 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.462194 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.462917 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.463022 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.463169 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:16.463343 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 04:34:16.463443 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":58,"threads_started":1}
I20260620 04:34:16.463887 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.457676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.0.98.3:38921)
0620 04:34:16.458747 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:16.458757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:16.458775 (+ 18us) server_negotiation.cc:407] Connection header received
0620 04:34:16.458833 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:16.458837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:16.458974 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:16.459233 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:16.459441 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.460607 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.462504 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.463253 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.463504 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.463636 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:16.463684 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 04:34:16.463769 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":98,"threads_started":1}
I20260620 04:34:16.463781 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.457553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38921 (local address 127.0.0.1:58494)
0620 04:34:16.458142 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:16.458189 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:16.458334 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:16.459117 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:16.459120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:16.459158 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:16.459328 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.459335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.460742 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.460745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.462385 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.462396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.463391 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.463394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.463518 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.463539 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:34:16.463597 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 04:34:16.463658 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":252,"spinlock_wait_cycles":1792,"thread_start_us":73,"threads_started":1}
I20260620 04:34:16.464133 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.458241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43457 (local address 127.0.0.1:43182)
0620 04:34:16.459005 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:16.459016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:16.459102 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:16.459742 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:16.459744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:16.459759 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:16.459818 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.459822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.460764 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.460767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.462386 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.462396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.463384 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.463387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.463870 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.463891 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:34:16.463955 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 04:34:16.464016 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":184,"thread_start_us":94,"threads_started":1}
I20260620 04:34:16.464525 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.458523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43182 (local address 127.0.98.2:43457)
0620 04:34:16.458902 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:16.458924 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:16.459190 (+ 266us) server_negotiation.cc:407] Connection header received
0620 04:34:16.459265 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:16.459269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:16.459479 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:16.459617 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:16.459929 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.460638 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.462525 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.463098 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.463192 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.463327 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:16.464271 (+ 944us) server_negotiation.cc:299] Negotiation successful
0620 04:34:16.464370 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":85,"threads_started":1}
I20260620 04:34:16.467271 19974 tablet_service.cc:1467] Processing CreateTablet for tablet 12e7960d584c48e2bf257c395ebc2c3c (DEFAULT_TABLE table=test-table [id=26fb05c6fbe246fd939d9b2fa6652930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:16.467281 19706 tablet_service.cc:1467] Processing CreateTablet for tablet 12e7960d584c48e2bf257c395ebc2c3c (DEFAULT_TABLE table=test-table [id=26fb05c6fbe246fd939d9b2fa6652930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:16.467280 19840 tablet_service.cc:1467] Processing CreateTablet for tablet 12e7960d584c48e2bf257c395ebc2c3c (DEFAULT_TABLE table=test-table [id=26fb05c6fbe246fd939d9b2fa6652930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:16.468259 19974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e7960d584c48e2bf257c395ebc2c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:16.468305 19706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e7960d584c48e2bf257c395ebc2c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:16.468300 19840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e7960d584c48e2bf257c395ebc2c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:16.474589 20057 tablet_bootstrap.cc:492] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: Bootstrap starting.
I20260620 04:34:16.476131 20058 tablet_bootstrap.cc:492] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263: Bootstrap starting.
I20260620 04:34:16.476179 20059 tablet_bootstrap.cc:492] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: Bootstrap starting.
I20260620 04:34:16.477561 20057 tablet_bootstrap.cc:654] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:16.477977 20058 tablet_bootstrap.cc:654] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:16.478685 20059 tablet_bootstrap.cc:654] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:16.478816 20057 log.cc:824] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:16.479033 20058 log.cc:824] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:16.479456 20059 log.cc:824] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:16.480865 20057 tablet_bootstrap.cc:492] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: No bootstrap required, opened a new log
I20260620 04:34:16.481108 20057 ts_tablet_manager.cc:1397] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 04:34:16.481393 20059 tablet_bootstrap.cc:492] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: No bootstrap required, opened a new log
I20260620 04:34:16.481583 20059 ts_tablet_manager.cc:1397] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:34:16.485116 20059 raft_consensus.cc:348] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 } } peers { permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 } } peers { permanent_uuid: "ae5396e2b76441808ceec06b043a4b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43457 } }
I20260620 04:34:16.485064 20057 raft_consensus.cc:348] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 } } peers { permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 } } peers { permanent_uuid: "ae5396e2b76441808ceec06b043a4b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43457 } }
I20260620 04:34:16.485373 20057 raft_consensus.cc:374] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:16.485373 20059 raft_consensus.cc:374] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:16.485433 20059 raft_consensus.cc:729] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5396e2b76441808ceec06b043a4b3a, State: Initialized, Role: FOLLOWER
I20260620 04:34:16.485464 20057 raft_consensus.cc:729] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c3b5da17ee4923901971134abf1ee5, State: Initialized, Role: FOLLOWER
I20260620 04:34:16.485936 20057 consensus_queue.cc:260] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 } } peers { permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 } } peers { permanent_uuid: "ae5396e2b76441808ceec06b043a4b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43457 } }
I20260620 04:34:16.485936 20059 consensus_queue.cc:260] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 } } peers { permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 } } peers { permanent_uuid: "ae5396e2b76441808ceec06b043a4b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43457 } }
I20260620 04:34:16.486584 20058 tablet_bootstrap.cc:492] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263: No bootstrap required, opened a new log
I20260620 04:34:16.486784 20058 ts_tablet_manager.cc:1397] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260620 04:34:16.486830 20040 heartbeater.cc:503] Master 127.0.98.62:35125 was elected leader, sending a full tablet report...
I20260620 04:34:16.486958 20059 ts_tablet_manager.cc:1428] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 04:34:16.486963 20057 ts_tablet_manager.cc:1428] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:34:16.487365 19906 heartbeater.cc:503] Master 127.0.98.62:35125 was elected leader, sending a full tablet report...
I20260620 04:34:16.490542 20058 raft_consensus.cc:348] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 } } peers { permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 } } peers { permanent_uuid: "ae5396e2b76441808ceec06b043a4b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43457 } }
I20260620 04:34:16.490837 20058 raft_consensus.cc:374] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:16.490895 20058 raft_consensus.cc:729] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4837953e856a40e090547aaae334e263, State: Initialized, Role: FOLLOWER
I20260620 04:34:16.491343 20058 consensus_queue.cc:260] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 } } peers { permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 } } peers { permanent_uuid: "ae5396e2b76441808ceec06b043a4b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43457 } }
I20260620 04:34:16.492075 19772 heartbeater.cc:503] Master 127.0.98.62:35125 was elected leader, sending a full tablet report...
I20260620 04:34:16.492419 20058 ts_tablet_manager.cc:1428] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 04:34:16.582527 20063 raft_consensus.cc:482] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:34:16.582718 20063 raft_consensus.cc:504] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 } } peers { permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 } } peers { permanent_uuid: "ae5396e2b76441808ceec06b043a4b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43457 } }
I20260620 04:34:16.583843 20063 leader_election.cc:288] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c3b5da17ee4923901971134abf1ee5 (127.0.98.3:38921), 4837953e856a40e090547aaae334e263 (127.0.98.1:44341)
I20260620 04:34:16.587355 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.584102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44341 (local address 127.0.98.2:47805)
0620 04:34:16.584245 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:16.584260 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:16.584351 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:16.584603 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:16.584606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:16.584623 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:16.584747 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.584752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.585510 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.585513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.586288 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.586297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.586956 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.586960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.587060 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.587078 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 04:34:16.587142 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 04:34:16.587215 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"spinlock_wait_cycles":15744}
I20260620 04:34:16.587355 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.584211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47805 (local address 127.0.98.1:44341)
0620 04:34:16.584325 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:16.584329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:16.584355 (+ 26us) server_negotiation.cc:407] Connection header received
0620 04:34:16.584469 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:16.584472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:16.584517 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:16.584579 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:16.584861 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.585404 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.586415 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.586841 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.586934 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.587000 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:16.587172 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 04:34:16.587239 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:34:16.587507 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.584226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38695 (local address 127.0.98.3:38921)
0620 04:34:16.584360 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:16.584364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:16.584643 (+ 279us) server_negotiation.cc:407] Connection header received
0620 04:34:16.584770 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:16.584773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:16.584817 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:16.584877 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:16.585149 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.585625 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.586613 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.587013 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.587101 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.587158 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:16.587333 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 04:34:16.587397 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 04:34:16.587522 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.584106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38921 (local address 127.0.98.2:38695)
0620 04:34:16.584550 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:16.584562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:16.584671 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:16.584896 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:16.584899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:16.584914 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:16.585045 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.585053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.585727 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.585730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.586497 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:16.586505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.587122 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.587125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:16.587224 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.587245 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 04:34:16.587307 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 04:34:16.587369 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":124,"thread_start_us":149,"threads_started":1}
I20260620 04:34:16.588188 19726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7960d584c48e2bf257c395ebc2c3c" candidate_uuid: "ae5396e2b76441808ceec06b043a4b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4837953e856a40e090547aaae334e263" is_pre_election: true
I20260620 04:34:16.588183 19994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7960d584c48e2bf257c395ebc2c3c" candidate_uuid: "ae5396e2b76441808ceec06b043a4b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c3b5da17ee4923901971134abf1ee5" is_pre_election: true
I20260620 04:34:16.588517 19994 raft_consensus.cc:2454] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5396e2b76441808ceec06b043a4b3a in term 0.
I20260620 04:34:16.588517 19726 raft_consensus.cc:2454] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5396e2b76441808ceec06b043a4b3a in term 0.
I20260620 04:34:16.589059 19793 leader_election.cc:302] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c3b5da17ee4923901971134abf1ee5, ae5396e2b76441808ceec06b043a4b3a; no voters:
I20260620 04:34:16.589345 20063 raft_consensus.cc:2781] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:34:16.589447 20063 raft_consensus.cc:482] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:34:16.589498 20063 raft_consensus.cc:3037] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:16.590521 20063 raft_consensus.cc:504] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 } } peers { permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 } } peers { permanent_uuid: "ae5396e2b76441808ceec06b043a4b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43457 } }
I20260620 04:34:16.590865 20063 leader_election.cc:288] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [CANDIDATE]: Term 1 election: Requested vote from peers 87c3b5da17ee4923901971134abf1ee5 (127.0.98.3:38921), 4837953e856a40e090547aaae334e263 (127.0.98.1:44341)
I20260620 04:34:16.591204 19994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7960d584c48e2bf257c395ebc2c3c" candidate_uuid: "ae5396e2b76441808ceec06b043a4b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c3b5da17ee4923901971134abf1ee5"
I20260620 04:34:16.591243 19726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e7960d584c48e2bf257c395ebc2c3c" candidate_uuid: "ae5396e2b76441808ceec06b043a4b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4837953e856a40e090547aaae334e263"
I20260620 04:34:16.591320 19994 raft_consensus.cc:3037] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:16.591351 19726 raft_consensus.cc:3037] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:16.592314 19726 raft_consensus.cc:2454] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5396e2b76441808ceec06b043a4b3a in term 1.
I20260620 04:34:16.592322 19994 raft_consensus.cc:2454] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5396e2b76441808ceec06b043a4b3a in term 1.
I20260620 04:34:16.592597 19793 leader_election.cc:302] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4837953e856a40e090547aaae334e263, 87c3b5da17ee4923901971134abf1ee5, ae5396e2b76441808ceec06b043a4b3a; no voters:
I20260620 04:34:16.592769 20063 raft_consensus.cc:2781] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:16.592981 20063 raft_consensus.cc:686] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 1 LEADER]: Becoming Leader. State: Replica: ae5396e2b76441808ceec06b043a4b3a, State: Running, Role: LEADER
I20260620 04:34:16.593312 20063 consensus_queue.cc:237] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 } } peers { permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 } } peers { permanent_uuid: "ae5396e2b76441808ceec06b043a4b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43457 } }
I20260620 04:34:16.596276 19578 catalog_manager.cc:5495] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to ae5396e2b76441808ceec06b043a4b3a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ae5396e2b76441808ceec06b043a4b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae5396e2b76441808ceec06b043a4b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43457 } health_report { overall_health: HEALTHY } } }
I20260620 04:34:16.618091 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.614941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.0.98.2:43457)
0620 04:34:16.615062 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:16.615065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:16.615094 (+ 29us) server_negotiation.cc:407] Connection header received
0620 04:34:16.615173 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:16.615175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:16.615228 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:16.615306 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:16.615523 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.616014 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.616770 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.616978 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.617068 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.617344 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:16.617782 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:16.617875 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:16.617925 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 04:34:16.617975 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:34:16.624332 19726 raft_consensus.cc:1261] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263 [term 1 FOLLOWER]: Refusing update from remote peer ae5396e2b76441808ceec06b043a4b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:16.624328 19994 raft_consensus.cc:1261] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5 [term 1 FOLLOWER]: Refusing update from remote peer ae5396e2b76441808ceec06b043a4b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:16.625038 20063 consensus_queue.cc:1035] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c3b5da17ee4923901971134abf1ee5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:16.625286 20067 consensus_queue.cc:1035] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4837953e856a40e090547aaae334e263" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:16.633526 20070 mvcc.cc:205] Tried to move back new op lower bound from 7298785511925280768 to 7298785511808544768. Current Snapshot: MvccSnapshot[applied={T|T < 7298785511925280768 or (T in {7298785511925280768})}]
I20260620 04:34:16.634382 20071 mvcc.cc:205] Tried to move back new op lower bound from 7298785511925280768 to 7298785511808544768. Current Snapshot: MvccSnapshot[applied={T|T < 7298785511925280768}]
I20260620 04:34:16.641130 19580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:16.641275 19580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:16.643532 19580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0079689aad24596a88033552fc51688: Sending DeleteTablet for 3 replicas of tablet 12e7960d584c48e2bf257c395ebc2c3c
I20260620 04:34:16.644279 19974 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e7960d584c48e2bf257c395ebc2c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58494
I20260620 04:34:16.644353 19840 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e7960d584c48e2bf257c395ebc2c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43182
I20260620 04:34:16.644733 20079 tablet_replica.cc:330] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: stopping tablet replica
I20260620 04:34:16.644750 19706 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e7960d584c48e2bf257c395ebc2c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44904
I20260620 04:34:16.645064 20079 raft_consensus.cc:2229] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:34:16.645408 20079 raft_consensus.cc:2258] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:16.645448 20081 tablet_replica.cc:330] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263: stopping tablet replica
I20260620 04:34:16.645766 20081 raft_consensus.cc:2229] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:34:16.645915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 19643
I20260620 04:34:16.646066 20081 raft_consensus.cc:2258] T 12e7960d584c48e2bf257c395ebc2c3c P 4837953e856a40e090547aaae334e263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:16.646611 20080 tablet_replica.cc:330] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: stopping tablet replica
I20260620 04:34:16.646850 20080 raft_consensus.cc:2229] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:34:16.647212 20080 raft_consensus.cc:2258] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:16.648227 20079 ts_tablet_manager.cc:1905] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:16.648553 20080 ts_tablet_manager.cc:1905] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:16.651103 20079 ts_tablet_manager.cc:1918] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:16.651135 20080 ts_tablet_manager.cc:1918] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:16.651214 20080 log.cc:1196] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-1/wal/wals/12e7960d584c48e2bf257c395ebc2c3c
I20260620 04:34:16.651239 20079 log.cc:1196] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/12e7960d584c48e2bf257c395ebc2c3c
I20260620 04:34:16.651575 20080 ts_tablet_manager.cc:1939] T 12e7960d584c48e2bf257c395ebc2c3c P ae5396e2b76441808ceec06b043a4b3a: Deleting consensus metadata
I20260620 04:34:16.651558 20079 ts_tablet_manager.cc:1939] T 12e7960d584c48e2bf257c395ebc2c3c P 87c3b5da17ee4923901971134abf1ee5: Deleting consensus metadata
I20260620 04:34:16.652796 19567 catalog_manager.cc:4841] TS ae5396e2b76441808ceec06b043a4b3a (127.0.98.2:43457): tablet 12e7960d584c48e2bf257c395ebc2c3c (table test-table [id=26fb05c6fbe246fd939d9b2fa6652930]) successfully deleted
I20260620 04:34:16.652797 19565 catalog_manager.cc:4841] TS 87c3b5da17ee4923901971134abf1ee5 (127.0.98.3:38921): tablet 12e7960d584c48e2bf257c395ebc2c3c (table test-table [id=26fb05c6fbe246fd939d9b2fa6652930]) successfully deleted
W20260620 04:34:16.653375 19567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44341 (error 108)
I20260620 04:34:16.653599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 19777
I20260620 04:34:16.654506 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.654207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44341 (local address 127.0.0.1:44906)
0620 04:34:16.654350 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:16.654424 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 04:34:16.654666 19567 catalog_manager.cc:4584] TS 4837953e856a40e090547aaae334e263 (127.0.98.1:44341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12e7960d584c48e2bf257c395ebc2c3c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44341: connect: Connection refused (error 111)
I20260620 04:34:16.658468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 19911
I20260620 04:34:16.663448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 19549
2026-06-20T04:34:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2202 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T04:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:16Z Disabled control of system clock
I20260620 04:34:16.690939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42393
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:34:16.798724 20086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:16.799005 20086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:16.799041 20086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:16.802982 20086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:16.803040 20086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:16.803054 20086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:16.803071 20086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:16.803084 20086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:16.807843 20086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:16.808874 20086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:16.810257 20086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:16.818457 20092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:16.818506 20086 server_base.cc:1034] running on GCE node
W20260620 04:34:16.818471 20091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:16.818471 20094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:16.819324 20086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:16.820448 20086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:16.821604 20086 hybrid_clock.cc:648] HybridClock initialized: now 1781930056821592 us; error 34 us; skew 500 ppm
I20260620 04:34:16.823498 20086 webserver.cc:460] Webserver started at http://127.0.98.62:38527/ using document root <none> and password file <none>
I20260620 04:34:16.824110 20086 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:16.824184 20086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:16.824390 20086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:16.826541 20086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b"
format_stamp: "Formatted at 2026-06-20 04:34:16 on dist-test-slave-rzqm"
server_key: "1742ce076460ca94aad6aef6a597e8ec"
server_key_iv: "f225294213a7b337f24b5df2a5130744"
server_key_version: "encryptionkey@0"
I20260620 04:34:16.827093 20086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b"
format_stamp: "Formatted at 2026-06-20 04:34:16 on dist-test-slave-rzqm"
server_key: "1742ce076460ca94aad6aef6a597e8ec"
server_key_iv: "f225294213a7b337f24b5df2a5130744"
server_key_version: "encryptionkey@0"
I20260620 04:34:16.830870 20086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:34:16.833272 20100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:16.834537 20086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:34:16.834656 20086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b"
format_stamp: "Formatted at 2026-06-20 04:34:16 on dist-test-slave-rzqm"
server_key: "1742ce076460ca94aad6aef6a597e8ec"
server_key_iv: "f225294213a7b337f24b5df2a5130744"
server_key_version: "encryptionkey@0"
I20260620 04:34:16.834756 20086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:16.856340 20086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:16.857095 20086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:16.857272 20086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:16.864027 20086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42393
I20260620 04:34:16.864043 20152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42393 every 8 connection(s)
I20260620 04:34:16.864984 20086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:16.866458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 20086
I20260620 04:34:16.866564 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:16.866817 392 external_mini_cluster.cc:1428] Setting key 3d68e42d4e4ae0be80fc84dc8fbdc2c6
I20260620 04:34:16.868422 20153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:16.874064 20153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b: Bootstrap starting.
I20260620 04:34:16.875859 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:16.868175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.0.98.62:42393)
0620 04:34:16.868877 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:16.868894 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:16.868933 (+ 39us) server_negotiation.cc:407] Connection header received
0620 04:34:16.869632 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:16.869649 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:16.870065 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:16.870380 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:16.870830 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.871799 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.872677 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:16.873193 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:16.873344 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:16.874156 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:16.874163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:16.874181 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:16.874579 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:16.874634 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:16.874656 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:16.874738 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:16.874879 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 04:34:16.875140 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":154,"threads_started":1}
I20260620 04:34:16.876353 20153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:16.877187 20153 log.cc:824] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:16.879138 20153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b: No bootstrap required, opened a new log
I20260620 04:34:16.882707 20153 raft_consensus.cc:348] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } }
I20260620 04:34:16.882949 20153 raft_consensus.cc:374] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:16.882992 20153 raft_consensus.cc:729] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89bef3f7b7f42dcb09fb4b1f4e66f4b, State: Initialized, Role: FOLLOWER
I20260620 04:34:16.883349 20153 consensus_queue.cc:260] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } }
I20260620 04:34:16.883481 20153 raft_consensus.cc:388] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:16.883530 20153 raft_consensus.cc:482] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:16.883599 20153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:16.884444 20153 raft_consensus.cc:504] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } }
I20260620 04:34:16.884789 20153 leader_election.cc:302] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c89bef3f7b7f42dcb09fb4b1f4e66f4b; no voters:
I20260620 04:34:16.885141 20153 leader_election.cc:288] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:16.885269 20158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:16.885499 20158 raft_consensus.cc:686] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [term 1 LEADER]: Becoming Leader. State: Replica: c89bef3f7b7f42dcb09fb4b1f4e66f4b, State: Running, Role: LEADER
I20260620 04:34:16.885792 20158 consensus_queue.cc:237] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } }
I20260620 04:34:16.886343 20153 sys_catalog.cc:564] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:16.887260 20159 sys_catalog.cc:455] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } } }
I20260620 04:34:16.887405 20159 sys_catalog.cc:458] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:16.887707 20160 sys_catalog.cc:455] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c89bef3f7b7f42dcb09fb4b1f4e66f4b. Latest consensus state: current_term: 1 leader_uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c89bef3f7b7f42dcb09fb4b1f4e66f4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42393 } } }
I20260620 04:34:16.887786 20167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:16.887806 20160 sys_catalog.cc:458] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:16.890926 20167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:16.896631 20167 catalog_manager.cc:1347] Generated new cluster ID: 509d4ed62f2c479bbc6dfda08605d53a
I20260620 04:34:16.896713 20167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:16.917377 20167 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:16.918354 20167 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:16.925391 20167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b: Generated new TSK 0
I20260620 04:34:16.926191 20167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:16.930574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42393
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 04:34:17.041200 20177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:17.041468 20177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:17.041508 20177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:17.045533 20177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:17.045614 20177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:17.045686 20177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 04:34:17.050422 20177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:17.051457 20177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:17.052775 20177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:17.062282 20185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:17.062278 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:17.062282 20183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:17.062517 20177 server_base.cc:1034] running on GCE node
I20260620 04:34:17.063025 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:17.063642 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:17.064873 20177 hybrid_clock.cc:648] HybridClock initialized: now 1781930057064842 us; error 41 us; skew 500 ppm
I20260620 04:34:17.066721 20177 webserver.cc:460] Webserver started at http://127.0.98.1:34919/ using document root <none> and password file <none>
I20260620 04:34:17.067291 20177 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:17.067359 20177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:17.067548 20177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:17.069685 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/data/instance:
uuid: "36c1261eb10842f38bc10174cc57dc7f"
format_stamp: "Formatted at 2026-06-20 04:34:17 on dist-test-slave-rzqm"
server_key: "1d4a9a847c7c0537fd931a37728803ec"
server_key_iv: "b04839d36bdd124470f7927ee5c7f664"
server_key_version: "encryptionkey@0"
I20260620 04:34:17.070214 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/wal/instance:
uuid: "36c1261eb10842f38bc10174cc57dc7f"
format_stamp: "Formatted at 2026-06-20 04:34:17 on dist-test-slave-rzqm"
server_key: "1d4a9a847c7c0537fd931a37728803ec"
server_key_iv: "b04839d36bdd124470f7927ee5c7f664"
server_key_version: "encryptionkey@0"
I20260620 04:34:17.073791 20177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:34:17.076221 20191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:17.077529 20177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:17.077637 20177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/wal
uuid: "36c1261eb10842f38bc10174cc57dc7f"
format_stamp: "Formatted at 2026-06-20 04:34:17 on dist-test-slave-rzqm"
server_key: "1d4a9a847c7c0537fd931a37728803ec"
server_key_iv: "b04839d36bdd124470f7927ee5c7f664"
server_key_version: "encryptionkey@0"
I20260620 04:34:17.077739 20177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:17.095530 20177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:17.096254 20177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:17.096442 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:17.097065 20177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:17.098011 20177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:17.098057 20177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:17.098109 20177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:17.098137 20177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:17.106789 20177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45177
I20260620 04:34:17.106812 20304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45177 every 8 connection(s)
I20260620 04:34:17.107738 20177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/data/info.pb
I20260620 04:34:17.116990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 20177
I20260620 04:34:17.117154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-0/wal/instance
I20260620 04:34:17.117432 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.109617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55047 (local address 127.0.98.62:42393)
0620 04:34:17.109773 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:17.109777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:17.110871 (+ 1094us) server_negotiation.cc:407] Connection header received
0620 04:34:17.112267 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:17.112271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:17.112325 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:17.112403 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:17.113379 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.114283 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.115455 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.115673 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.115754 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.116873 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:17.116875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:17.116878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:17.116890 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:17.116909 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:17.116916 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:17.116980 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:17.117224 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 04:34:17.117311 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 04:34:17.117493 392 external_mini_cluster.cc:1428] Setting key 3760b0ae56562f1dd7b9301d58a229c6
I20260620 04:34:17.118389 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.110045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42393 (local address 127.0.98.1:55047)
0620 04:34:17.110712 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:17.110767 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:17.112032 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:17.112569 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:17.112576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:17.112665 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:17.113201 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.113214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.114416 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.114420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.115338 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.115348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.115793 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.115796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.115976 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.116581 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:34:17.116582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:17.116624 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:34:17.116719 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:17.117019 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:17.117029 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:17.117035 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:34:17.117206 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 04:34:17.117514 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20260620 04:34:17.119551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42393
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:34:17.119613 20305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42393
I20260620 04:34:17.119856 20305 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:17.120368 20305 heartbeater.cc:511] Master 127.0.98.62:42393 requested a full tablet report, sending...
I20260620 04:34:17.121866 20114 ts_manager.cc:194] Registered new tserver with Master: 36c1261eb10842f38bc10174cc57dc7f (127.0.98.1:45177)
I20260620 04:34:17.122999 20114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55047
W20260620 04:34:17.228895 20308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:17.229219 20308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:17.229275 20308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:17.233810 20308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:17.233883 20308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:17.233956 20308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 04:34:17.240283 20308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:17.241461 20308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:17.242841 20308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:17.252415 20314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:17.252432 20316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:17.252686 20308 server_base.cc:1034] running on GCE node
W20260620 04:34:17.252415 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:17.253175 20308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:17.253882 20308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:17.255059 20308 hybrid_clock.cc:648] HybridClock initialized: now 1781930057255043 us; error 25 us; skew 500 ppm
I20260620 04:34:17.257057 20308 webserver.cc:460] Webserver started at http://127.0.98.2:40797/ using document root <none> and password file <none>
I20260620 04:34:17.257628 20308 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:17.257692 20308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:17.257901 20308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:17.260130 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/data/instance:
uuid: "6a27ae8ec02a477cb779e7663130b369"
format_stamp: "Formatted at 2026-06-20 04:34:17 on dist-test-slave-rzqm"
server_key: "e24705c7281fee0f2be514f77d06f817"
server_key_iv: "1385acedeb2af7a12298007bd38cc7a7"
server_key_version: "encryptionkey@0"
I20260620 04:34:17.260670 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6a27ae8ec02a477cb779e7663130b369"
format_stamp: "Formatted at 2026-06-20 04:34:17 on dist-test-slave-rzqm"
server_key: "e24705c7281fee0f2be514f77d06f817"
server_key_iv: "1385acedeb2af7a12298007bd38cc7a7"
server_key_version: "encryptionkey@0"
I20260620 04:34:17.264406 20308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:34:17.267009 20322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:17.268369 20308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 04:34:17.268500 20308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/wal
uuid: "6a27ae8ec02a477cb779e7663130b369"
format_stamp: "Formatted at 2026-06-20 04:34:17 on dist-test-slave-rzqm"
server_key: "e24705c7281fee0f2be514f77d06f817"
server_key_iv: "1385acedeb2af7a12298007bd38cc7a7"
server_key_version: "encryptionkey@0"
I20260620 04:34:17.268616 20308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:17.304730 20308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:17.305524 20308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:17.305698 20308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:17.306313 20308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:17.307273 20308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:17.307324 20308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:17.307370 20308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:17.307399 20308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:17.315623 20308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35741
I20260620 04:34:17.315652 20435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35741 every 8 connection(s)
I20260620 04:34:17.316540 20308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/data/info.pb
I20260620 04:34:17.326139 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.318474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37547 (local address 127.0.98.62:42393)
0620 04:34:17.318604 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:17.318607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:17.319648 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 04:34:17.321111 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:17.321115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:17.321203 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:17.321428 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:17.322460 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.322980 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.324172 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.324388 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.324484 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.325664 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:17.325667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:17.325669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:17.325682 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:17.325703 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:17.325711 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:17.325772 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:17.325915 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 04:34:17.325987 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:34:17.326483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 20308
I20260620 04:34:17.326617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-1/wal/instance
I20260620 04:34:17.326929 392 external_mini_cluster.cc:1428] Setting key c86d2fed0235c42501cf3edd572cd23d
I20260620 04:34:17.327361 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.318814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42393 (local address 127.0.98.2:37547)
0620 04:34:17.319489 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:17.319533 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:17.320831 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:17.321610 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:17.321621 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:17.321726 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:17.322254 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.322269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.323105 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.323108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.324047 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.324057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.324544 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.324548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.324748 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.325381 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:34:17.325383 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:17.325428 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:34:17.325498 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:17.325784 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:17.325794 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:17.325800 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 04:34:17.325915 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 04:34:17.326224 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260620 04:34:17.328513 20436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42393
I20260620 04:34:17.328771 20436 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:17.328964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42393
--builtin_ntp_servers=127.0.98.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 04:34:17.329437 20436 heartbeater.cc:511] Master 127.0.98.62:42393 requested a full tablet report, sending...
I20260620 04:34:17.330642 20114 ts_manager.cc:194] Registered new tserver with Master: 6a27ae8ec02a477cb779e7663130b369 (127.0.98.2:35741)
I20260620 04:34:17.331450 20114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37547
W20260620 04:34:17.439298 20439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:17.439592 20439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:17.439633 20439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:17.443653 20439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:17.443712 20439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:17.443769 20439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 04:34:17.448438 20439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:17.449512 20439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:17.450946 20439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:17.460729 20444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:17.460727 20445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:17.460777 20439 server_base.cc:1034] running on GCE node
W20260620 04:34:17.460738 20447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:17.461401 20439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:17.462015 20439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:17.463212 20439 hybrid_clock.cc:648] HybridClock initialized: now 1781930057463187 us; error 34 us; skew 500 ppm
I20260620 04:34:17.465075 20439 webserver.cc:460] Webserver started at http://127.0.98.3:40865/ using document root <none> and password file <none>
I20260620 04:34:17.465632 20439 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:17.465699 20439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:17.465902 20439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:17.468014 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/data/instance:
uuid: "0c65cfd261ef494597fa9fc6b48c9fd8"
format_stamp: "Formatted at 2026-06-20 04:34:17 on dist-test-slave-rzqm"
server_key: "861a156a7b026c8202cbc9b159a86993"
server_key_iv: "3c189acec28addb685d000f12aa6d890"
server_key_version: "encryptionkey@0"
I20260620 04:34:17.468542 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0c65cfd261ef494597fa9fc6b48c9fd8"
format_stamp: "Formatted at 2026-06-20 04:34:17 on dist-test-slave-rzqm"
server_key: "861a156a7b026c8202cbc9b159a86993"
server_key_iv: "3c189acec28addb685d000f12aa6d890"
server_key_version: "encryptionkey@0"
I20260620 04:34:17.472190 20439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:34:17.474673 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:17.475982 20439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:17.476089 20439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/wal
uuid: "0c65cfd261ef494597fa9fc6b48c9fd8"
format_stamp: "Formatted at 2026-06-20 04:34:17 on dist-test-slave-rzqm"
server_key: "861a156a7b026c8202cbc9b159a86993"
server_key_iv: "3c189acec28addb685d000f12aa6d890"
server_key_version: "encryptionkey@0"
I20260620 04:34:17.476186 20439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:17.495283 20439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:17.496001 20439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:17.496171 20439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:17.496742 20439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 04:34:17.497731 20439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 04:34:17.497779 20439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:17.497833 20439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 04:34:17.497869 20439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:17.506382 20439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43259
I20260620 04:34:17.506453 20566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43259 every 8 connection(s)
I20260620 04:34:17.507293 20439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/data/info.pb
I20260620 04:34:17.515692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 20439
I20260620 04:34:17.515825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/wal/instance
I20260620 04:34:17.516162 392 external_mini_cluster.cc:1428] Setting key ac303f40512846a828e1e39b738243b9
I20260620 04:34:17.518078 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.509163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39805 (local address 127.0.98.62:42393)
0620 04:34:17.509318 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:17.509323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:17.510401 (+ 1078us) server_negotiation.cc:407] Connection header received
0620 04:34:17.511764 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:17.511767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:17.511816 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:17.511899 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:17.512792 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.513371 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.514570 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.514887 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.515000 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.517507 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:17.517511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:17.517514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:17.517530 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:17.517557 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:17.517568 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:17.517635 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:17.517859 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 04:34:17.517935 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 04:34:17.519276 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.509601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42393 (local address 127.0.98.3:39805)
0620 04:34:17.510256 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:17.510297 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:17.511563 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:17.512067 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:17.512074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:17.512162 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:17.512668 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.512681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.513469 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.513471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.514439 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.514449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.515069 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.515074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.515288 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.517107 (+ 1819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 04:34:17.517109 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:17.517175 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 04:34:17.517273 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:17.517674 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:17.517683 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:17.517690 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 04:34:17.517828 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 04:34:17.518166 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260620 04:34:17.520653 20567 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42393
I20260620 04:34:17.520936 20567 heartbeater.cc:464] Registering TS with master...
I20260620 04:34:17.521710 20567 heartbeater.cc:511] Master 127.0.98.62:42393 requested a full tablet report, sending...
I20260620 04:34:17.523008 20114 ts_manager.cc:194] Registered new tserver with Master: 0c65cfd261ef494597fa9fc6b48c9fd8 (127.0.98.3:43259)
I20260620 04:34:17.523691 20114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39805
I20260620 04:34:17.530427 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 04:34:17.540771 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.536193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.0.98.62:42393)
0620 04:34:17.536379 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:17.536383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:17.536557 (+ 174us) server_negotiation.cc:407] Connection header received
0620 04:34:17.536715 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:17.536719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:17.536771 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:17.536850 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:17.537219 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.537978 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.539086 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.539333 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.539446 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.540290 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:17.540294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:17.540297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:17.540312 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:17.540335 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:17.540344 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:17.540402 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:17.540546 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 04:34:17.540611 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 04:34:17.544546 20114 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 04:34:17.548060 20114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 04:34:17.563407 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.557156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35741 (local address 127.0.0.1:48938)
0620 04:34:17.557550 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:17.557604 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:17.557724 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:17.558724 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:17.558730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:17.558754 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:17.558903 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.558910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.560506 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.560509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.561798 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.561809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.562878 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.562882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.562994 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.563037 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 04:34:17.563120 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:34:17.563250 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":5,"thread_start_us":113,"threads_started":1}
I20260620 04:34:17.563773 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.557359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45177 (local address 127.0.0.1:56978)
0620 04:34:17.558006 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:17.558019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:17.558108 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:17.559244 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:17.559247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:17.559265 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:17.559367 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.559374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.560671 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.560675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.561935 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.561947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.563361 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.563365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.563462 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.563489 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 04:34:17.563560 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 04:34:17.563629 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":105,"threads_started":1}
I20260620 04:34:17.564360 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.558390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.0.98.1:45177)
0620 04:34:17.558789 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:17.558799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:17.558820 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:34:17.558902 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:17.558907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:17.559096 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:17.559251 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:17.559487 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.560555 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.562082 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.562949 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.563056 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.563294 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:17.564069 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 04:34:17.564196 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":122,"threads_started":1}
I20260620 04:34:17.564875 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.557895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.0.98.2:35741)
0620 04:34:17.558194 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:17.558206 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:17.558230 (+ 24us) server_negotiation.cc:407] Connection header received
0620 04:34:17.558309 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:17.558314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:17.558547 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:17.559055 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:17.559194 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.560369 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.561944 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.562753 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.564215 (+ 1462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.564567 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:17.564620 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 04:34:17.564720 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":125,"threads_started":1}
I20260620 04:34:17.564153 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.557531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43259 (local address 127.0.0.1:57946)
0620 04:34:17.558280 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:17.558292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:17.558367 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:17.559432 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:17.559435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:17.559451 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:17.559527 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.559531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.560466 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.560469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.562542 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.562553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.563782 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.563786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.563872 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.563894 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 04:34:17.563957 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 04:34:17.564024 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":95,"thread_start_us":94,"threads_started":1}
I20260620 04:34:17.565418 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.557978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.0.98.3:43259)
0620 04:34:17.558910 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:17.558924 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:17.558945 (+ 21us) server_negotiation.cc:407] Connection header received
0620 04:34:17.559006 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:17.559010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:17.559154 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:17.559301 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:17.559623 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.560358 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.562679 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.563429 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.563523 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.563709 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:17.565164 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0620 04:34:17.565264 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":163,"threads_started":1}
I20260620 04:34:17.567494 20239 tablet_service.cc:1467] Processing CreateTablet for tablet e03e4c2ffe0441e1bb6e48bfc7db2a68 (DEFAULT_TABLE table=test-table [id=2d2ee833e3644aca8bb4995b6fd18028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:17.567830 20501 tablet_service.cc:1467] Processing CreateTablet for tablet e03e4c2ffe0441e1bb6e48bfc7db2a68 (DEFAULT_TABLE table=test-table [id=2d2ee833e3644aca8bb4995b6fd18028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:17.567830 20370 tablet_service.cc:1467] Processing CreateTablet for tablet e03e4c2ffe0441e1bb6e48bfc7db2a68 (DEFAULT_TABLE table=test-table [id=2d2ee833e3644aca8bb4995b6fd18028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 04:34:17.568562 20239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03e4c2ffe0441e1bb6e48bfc7db2a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:17.568848 20501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03e4c2ffe0441e1bb6e48bfc7db2a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:17.568848 20370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03e4c2ffe0441e1bb6e48bfc7db2a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:17.575585 20584 tablet_bootstrap.cc:492] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f: Bootstrap starting.
I20260620 04:34:17.577409 20585 tablet_bootstrap.cc:492] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369: Bootstrap starting.
I20260620 04:34:17.577868 20584 tablet_bootstrap.cc:654] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:17.578712 20584 log.cc:824] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:17.579432 20585 tablet_bootstrap.cc:654] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:17.580300 20585 log.cc:824] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:17.580677 20584 tablet_bootstrap.cc:492] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f: No bootstrap required, opened a new log
I20260620 04:34:17.580854 20584 ts_tablet_manager.cc:1397] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 04:34:17.582448 20585 tablet_bootstrap.cc:492] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369: No bootstrap required, opened a new log
I20260620 04:34:17.582640 20585 ts_tablet_manager.cc:1397] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 04:34:17.585053 20584 raft_consensus.cc:348] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a27ae8ec02a477cb779e7663130b369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35741 } } peers { permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 } } peers { permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } }
I20260620 04:34:17.585323 20584 raft_consensus.cc:374] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:17.585376 20584 raft_consensus.cc:729] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c1261eb10842f38bc10174cc57dc7f, State: Initialized, Role: FOLLOWER
I20260620 04:34:17.585808 20584 consensus_queue.cc:260] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a27ae8ec02a477cb779e7663130b369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35741 } } peers { permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 } } peers { permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } }
I20260620 04:34:17.586694 20305 heartbeater.cc:503] Master 127.0.98.62:42393 was elected leader, sending a full tablet report...
I20260620 04:34:17.586884 20584 ts_tablet_manager.cc:1428] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 04:34:17.586882 20586 tablet_bootstrap.cc:492] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: Bootstrap starting.
I20260620 04:34:17.588698 20585 raft_consensus.cc:348] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a27ae8ec02a477cb779e7663130b369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35741 } } peers { permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 } } peers { permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } }
I20260620 04:34:17.589175 20585 raft_consensus.cc:374] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:17.589244 20585 raft_consensus.cc:729] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a27ae8ec02a477cb779e7663130b369, State: Initialized, Role: FOLLOWER
I20260620 04:34:17.589378 20586 tablet_bootstrap.cc:654] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:17.589748 20585 consensus_queue.cc:260] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a27ae8ec02a477cb779e7663130b369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35741 } } peers { permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 } } peers { permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } }
I20260620 04:34:17.590302 20586 log.cc:824] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:17.590921 20585 ts_tablet_manager.cc:1428] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20260620 04:34:17.591265 20436 heartbeater.cc:503] Master 127.0.98.62:42393 was elected leader, sending a full tablet report...
I20260620 04:34:17.593197 20586 tablet_bootstrap.cc:492] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: No bootstrap required, opened a new log
I20260620 04:34:17.593379 20586 ts_tablet_manager.cc:1397] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 04:34:17.596797 20586 raft_consensus.cc:348] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a27ae8ec02a477cb779e7663130b369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35741 } } peers { permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 } } peers { permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } }
I20260620 04:34:17.597086 20586 raft_consensus.cc:374] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:17.597151 20586 raft_consensus.cc:729] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c65cfd261ef494597fa9fc6b48c9fd8, State: Initialized, Role: FOLLOWER
I20260620 04:34:17.597558 20586 consensus_queue.cc:260] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a27ae8ec02a477cb779e7663130b369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35741 } } peers { permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 } } peers { permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } }
I20260620 04:34:17.598186 20567 heartbeater.cc:503] Master 127.0.98.62:42393 was elected leader, sending a full tablet report...
I20260620 04:34:17.598526 20586 ts_tablet_manager.cc:1428] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 04:34:17.608914 20306 tablet.cc:2369] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:34:17.662010 20590 raft_consensus.cc:482] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 04:34:17.662206 20590 raft_consensus.cc:504] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a27ae8ec02a477cb779e7663130b369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35741 } } peers { permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 } } peers { permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } }
I20260620 04:34:17.663352 20590 leader_election.cc:288] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36c1261eb10842f38bc10174cc57dc7f (127.0.98.1:45177), 0c65cfd261ef494597fa9fc6b48c9fd8 (127.0.98.3:43259)
I20260620 04:34:17.667680 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.663836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49075 (local address 127.0.98.3:43259)
0620 04:34:17.663948 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:17.663953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:17.664157 (+ 204us) server_negotiation.cc:407] Connection header received
0620 04:34:17.664252 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:17.664255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:17.664303 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:17.664376 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:17.664588 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.665290 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.666606 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.667211 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.667300 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.667340 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:17.667503 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 04:34:17.667568 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 04:34:17.667680 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.663665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58325 (local address 127.0.98.1:45177)
0620 04:34:17.663792 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:17.663796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:17.663813 (+ 17us) server_negotiation.cc:407] Connection header received
0620 04:34:17.663892 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:17.663895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:17.663947 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:17.664005 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 04:34:17.664274 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.665048 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.666491 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.667064 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.667171 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.667235 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:17.667476 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 04:34:17.667556 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:34:17.667672 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.663573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45177 (local address 127.0.98.2:58325)
0620 04:34:17.663679 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:17.663695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:17.663786 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:17.664028 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:17.664031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:17.664049 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:17.664173 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.664180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.665146 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.665148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.666371 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.666383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.667210 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.667213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.667329 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.667355 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 04:34:17.667441 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 04:34:17.667523 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 04:34:17.667678 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.663621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43259 (local address 127.0.98.2:49075)
0620 04:34:17.664060 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:17.664073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:17.664162 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:17.664387 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:17.664389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:17.664402 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 04:34:17.664494 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.664499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.665378 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.665381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.666506 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:17.666514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.667331 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.667334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:17.667415 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.667434 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 04:34:17.667489 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 04:34:17.667541 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":128,"threads_started":1}
I20260620 04:34:17.668375 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03e4c2ffe0441e1bb6e48bfc7db2a68" candidate_uuid: "6a27ae8ec02a477cb779e7663130b369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36c1261eb10842f38bc10174cc57dc7f" is_pre_election: true
I20260620 04:34:17.668418 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03e4c2ffe0441e1bb6e48bfc7db2a68" candidate_uuid: "6a27ae8ec02a477cb779e7663130b369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" is_pre_election: true
I20260620 04:34:17.668663 20259 raft_consensus.cc:2454] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a27ae8ec02a477cb779e7663130b369 in term 0.
I20260620 04:34:17.668671 20521 raft_consensus.cc:2454] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a27ae8ec02a477cb779e7663130b369 in term 0.
I20260620 04:34:17.669155 20324 leader_election.cc:302] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c65cfd261ef494597fa9fc6b48c9fd8, 36c1261eb10842f38bc10174cc57dc7f, 6a27ae8ec02a477cb779e7663130b369; no voters:
I20260620 04:34:17.669422 20590 raft_consensus.cc:2781] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 04:34:17.669500 20590 raft_consensus.cc:482] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 04:34:17.669543 20590 raft_consensus.cc:3037] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:17.670559 20590 raft_consensus.cc:504] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a27ae8ec02a477cb779e7663130b369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35741 } } peers { permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 } } peers { permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } }
I20260620 04:34:17.670907 20590 leader_election.cc:288] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [CANDIDATE]: Term 1 election: Requested vote from peers 36c1261eb10842f38bc10174cc57dc7f (127.0.98.1:45177), 0c65cfd261ef494597fa9fc6b48c9fd8 (127.0.98.3:43259)
I20260620 04:34:17.671277 20259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03e4c2ffe0441e1bb6e48bfc7db2a68" candidate_uuid: "6a27ae8ec02a477cb779e7663130b369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36c1261eb10842f38bc10174cc57dc7f"
I20260620 04:34:17.671311 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03e4c2ffe0441e1bb6e48bfc7db2a68" candidate_uuid: "6a27ae8ec02a477cb779e7663130b369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8"
I20260620 04:34:17.671401 20259 raft_consensus.cc:3037] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:17.671419 20521 raft_consensus.cc:3037] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:17.672444 20259 raft_consensus.cc:2454] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a27ae8ec02a477cb779e7663130b369 in term 1.
I20260620 04:34:17.672531 20521 raft_consensus.cc:2454] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a27ae8ec02a477cb779e7663130b369 in term 1.
I20260620 04:34:17.672778 20326 leader_election.cc:302] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36c1261eb10842f38bc10174cc57dc7f, 6a27ae8ec02a477cb779e7663130b369; no voters:
I20260620 04:34:17.672962 20590 raft_consensus.cc:2781] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:17.673214 20590 raft_consensus.cc:686] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 1 LEADER]: Becoming Leader. State: Replica: 6a27ae8ec02a477cb779e7663130b369, State: Running, Role: LEADER
I20260620 04:34:17.673518 20590 consensus_queue.cc:237] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a27ae8ec02a477cb779e7663130b369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35741 } } peers { permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 } } peers { permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } }
I20260620 04:34:17.676697 20114 catalog_manager.cc:5495] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a27ae8ec02a477cb779e7663130b369 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6a27ae8ec02a477cb779e7663130b369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a27ae8ec02a477cb779e7663130b369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } health_report { overall_health: UNKNOWN } } }
I20260620 04:34:17.686887 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.683515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.0.98.2:35741)
0620 04:34:17.683635 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:17.683639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:17.683653 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:34:17.683743 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:17.683746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:17.683796 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:17.683932 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:17.684157 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.684644 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.685507 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:17.685715 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:17.685828 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:17.686073 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:17.686475 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:17.686581 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:17.686675 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 04:34:17.686725 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:34:17.692461 20521 raft_consensus.cc:1261] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8 [term 1 FOLLOWER]: Refusing update from remote peer 6a27ae8ec02a477cb779e7663130b369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:17.692468 20259 raft_consensus.cc:1261] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f [term 1 FOLLOWER]: Refusing update from remote peer 6a27ae8ec02a477cb779e7663130b369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:17.693147 20594 consensus_queue.cc:1035] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c65cfd261ef494597fa9fc6b48c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:17.693313 20590 consensus_queue.cc:1035] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36c1261eb10842f38bc10174cc57dc7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:17.703238 20598 mvcc.cc:205] Tried to move back new op lower bound from 7298785516301303808 to 7298785516233261056. Current Snapshot: MvccSnapshot[applied={T|T < 7298785516301303808}]
I20260620 04:34:17.707679 20114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:17.707861 20114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 04:34:17.710222 20114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c89bef3f7b7f42dcb09fb4b1f4e66f4b: Sending DeleteTablet for 3 replicas of tablet e03e4c2ffe0441e1bb6e48bfc7db2a68
I20260620 04:34:17.710958 20370 tablet_service.cc:1514] Processing DeleteTablet for tablet e03e4c2ffe0441e1bb6e48bfc7db2a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:17 UTC) from {username='slave'} at 127.0.0.1:48938
I20260620 04:34:17.710963 20501 tablet_service.cc:1514] Processing DeleteTablet for tablet e03e4c2ffe0441e1bb6e48bfc7db2a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:17 UTC) from {username='slave'} at 127.0.0.1:57946
I20260620 04:34:17.711503 20608 tablet_replica.cc:330] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: stopping tablet replica
I20260620 04:34:17.711472 20239 tablet_service.cc:1514] Processing DeleteTablet for tablet e03e4c2ffe0441e1bb6e48bfc7db2a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 04:34:17 UTC) from {username='slave'} at 127.0.0.1:56978
I20260620 04:34:17.711848 20608 raft_consensus.cc:2229] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:34:17.711905 20609 tablet_replica.cc:330] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f: stopping tablet replica
I20260620 04:34:17.712195 20608 raft_consensus.cc:2258] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:17.712373 20609 raft_consensus.cc:2229] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 04:34:17.712581 20607 tablet_replica.cc:330] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369: stopping tablet replica
I20260620 04:34:17.712769 20609 raft_consensus.cc:2258] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 36c1261eb10842f38bc10174cc57dc7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:17.712818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 20177
I20260620 04:34:17.712848 20607 raft_consensus.cc:2229] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 1 LEADER]: Raft consensus shutting down.
I20260620 04:34:17.713229 20607 raft_consensus.cc:2258] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 04:34:17.714208 20608 ts_tablet_manager.cc:1905] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:17.716991 20608 ts_tablet_manager.cc:1918] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 04:34:17.717114 20608 log.cc:1196] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: Deleting WAL directory at /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781929807654528-392-0/minicluster-data/ts-2/wal/wals/e03e4c2ffe0441e1bb6e48bfc7db2a68
I20260620 04:34:17.717403 20608 ts_tablet_manager.cc:1939] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 0c65cfd261ef494597fa9fc6b48c9fd8: Deleting consensus metadata
I20260620 04:34:17.718660 20607 ts_tablet_manager.cc:1905] T e03e4c2ffe0441e1bb6e48bfc7db2a68 P 6a27ae8ec02a477cb779e7663130b369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 04:34:17.718680 20102 catalog_manager.cc:4841] TS 0c65cfd261ef494597fa9fc6b48c9fd8 (127.0.98.3:43259): tablet e03e4c2ffe0441e1bb6e48bfc7db2a68 (table test-table [id=2d2ee833e3644aca8bb4995b6fd18028]) successfully deleted
W20260620 04:34:17.720185 20104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45177 (error 108)
I20260620 04:34:17.721289 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.720957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45177 (local address 127.0.0.1:56982)
0620 04:34:17.721108 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:17.721180 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 04:34:17.721344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 20308
W20260620 04:34:17.722669 20104 catalog_manager.cc:4584] TS 36c1261eb10842f38bc10174cc57dc7f (127.0.98.1:45177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e03e4c2ffe0441e1bb6e48bfc7db2a68: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45177: connect: Connection refused (error 111)
I20260620 04:34:17.726440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 20439
I20260620 04:34:17.728019 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:17.726653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35741 (local address 127.0.0.1:48964)
0620 04:34:17.727885 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:17.727943 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1139}
I20260620 04:34:17.731451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 20086
2026-06-20T04:34:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1067 ms)
[----------] 6 tests from EncryptionPolicyTest (9822 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:17 dist-test-slave-rzqm krb5kdc[20613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:17 dist-test-slave-rzqm krb5kdc[20613](info): set up 2 sockets
Jun 20 04:34:17 dist-test-slave-rzqm krb5kdc[20613](info): commencing operation
krb5kdc: starting...
W20260620 04:34:18.692821 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:18 dist-test-slave-rzqm krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:18.807575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36637
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:18.917142 20626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:18.917421 20626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:18.917466 20626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:18.921525 20626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:18.921597 20626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:18.921617 20626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:18.921633 20626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:18.921645 20626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:18.927342 20626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36637
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:18.928484 20626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:18.929632 20626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:18.938481 20634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:18.938462 20632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:18.938450 20631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:18.938524 20626 server_base.cc:1034] running on GCE node
I20260620 04:34:18.939210 20626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:18.940369 20626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:18.941562 20626 hybrid_clock.cc:648] HybridClock initialized: now 1781930058941538 us; error 39 us; skew 500 ppm
Jun 20 04:34:18 dist-test-slave-rzqm krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:18.944355 20626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:18.945267 20626 webserver.cc:460] Webserver started at http://127.0.0.1:42627/ using document root <none> and password file <none>
I20260620 04:34:18.945799 20626 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:18.945858 20626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:18.946035 20626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:18.948062 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "90fbb4cbc20a4b988204f69e69e0f016"
format_stamp: "Formatted at 2026-06-20 04:34:18 on dist-test-slave-rzqm"
server_key: "8cfb9a17b24084a9c1fa227b78dce634"
server_key_iv: "9db58764fd2e83230644cff399bbcb18"
server_key_version: "encryptionkey@0"
I20260620 04:34:18.948588 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "90fbb4cbc20a4b988204f69e69e0f016"
format_stamp: "Formatted at 2026-06-20 04:34:18 on dist-test-slave-rzqm"
server_key: "8cfb9a17b24084a9c1fa227b78dce634"
server_key_iv: "9db58764fd2e83230644cff399bbcb18"
server_key_version: "encryptionkey@0"
I20260620 04:34:18.952699 20626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 04:34:18.955646 20641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:18.957129 20626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:18.957253 20626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "90fbb4cbc20a4b988204f69e69e0f016"
format_stamp: "Formatted at 2026-06-20 04:34:18 on dist-test-slave-rzqm"
server_key: "8cfb9a17b24084a9c1fa227b78dce634"
server_key_iv: "9db58764fd2e83230644cff399bbcb18"
server_key_version: "encryptionkey@0"
I20260620 04:34:18.957365 20626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:18.966275 20626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:18.969345 20626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:18.969506 20626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:18.976213 20626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36637
I20260620 04:34:18.976234 20693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36637 every 8 connection(s)
I20260620 04:34:18.977205 20626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:18.979946 20694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:18.983819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 20626
I20260620 04:34:18.983961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:18.984210 392 external_mini_cluster.cc:1428] Setting key a6d1b03d986aae83ebd0085152f6cc1e
I20260620 04:34:18.985668 20694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016: Bootstrap starting.
I20260620 04:34:18.988327 20694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:18.989099 20694 log.cc:824] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:18.990907 20694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016: No bootstrap required, opened a new log
Jun 20 04:34:18 dist-test-slave-rzqm krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:34:18.994571 20694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fbb4cbc20a4b988204f69e69e0f016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36637 } }
I20260620 04:34:18.994808 20694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:18.994884 20694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90fbb4cbc20a4b988204f69e69e0f016, State: Initialized, Role: FOLLOWER
I20260620 04:34:18.995298 20694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fbb4cbc20a4b988204f69e69e0f016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36637 } }
I20260620 04:34:18.995424 20694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:18.995497 20694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:18.995594 20694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:18.996459 20694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fbb4cbc20a4b988204f69e69e0f016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36637 } }
I20260620 04:34:18.996811 20694 leader_election.cc:302] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90fbb4cbc20a4b988204f69e69e0f016; no voters:
I20260620 04:34:18.997182 20694 leader_election.cc:288] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:18.997325 20699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:18.997560 20699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [term 1 LEADER]: Becoming Leader. State: Replica: 90fbb4cbc20a4b988204f69e69e0f016, State: Running, Role: LEADER
I20260620 04:34:18.997860 20699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fbb4cbc20a4b988204f69e69e0f016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36637 } }
I20260620 04:34:18.998380 20694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:18.998385 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:18.985658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59797 (local address 127.0.0.1:36637)
0620 04:34:18.986078 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:18.986097 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:18.986139 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:34:18.986992 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:18.987017 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:18.987443 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:18.987805 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:18.988495 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:18.989552 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:18.990434 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:18.990913 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:18.991063 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:18.993417 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:18.993423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:18.993437 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:18.993465 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:18.995212 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:18.995733 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:18.995739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:18.995745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:18.995851 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:18.996163 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:18.996166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:18.996167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:18.996561 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:18.996757 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:18.997161 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 04:34:18.997406 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":123,"threads_started":1}
I20260620 04:34:18.999054 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90fbb4cbc20a4b988204f69e69e0f016. Latest consensus state: current_term: 1 leader_uuid: "90fbb4cbc20a4b988204f69e69e0f016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fbb4cbc20a4b988204f69e69e0f016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36637 } } }
I20260620 04:34:18.999162 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:18.999724 20700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90fbb4cbc20a4b988204f69e69e0f016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fbb4cbc20a4b988204f69e69e0f016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36637 } } }
I20260620 04:34:18.999804 20700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:19.000070 20707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:19.001632 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:19.003858 20707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:19.010105 20707 catalog_manager.cc:1347] Generated new cluster ID: db51b3ef21b3488393086c0529b008a1
I20260620 04:34:19.010179 20707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:19.011799 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:19.003876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48359 (local address 127.0.0.1:36637)
0620 04:34:19.003992 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:19.003997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:19.004011 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:34:19.004138 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:19.004141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:19.004187 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:19.004268 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:19.005026 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:19.005728 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:19.006511 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:19.006758 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:19.006842 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:19.008191 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:19.008195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:19.008198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:19.008223 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:19.010132 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:19.010618 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:19.010621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:19.010623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:19.010700 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:19.011006 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:19.011009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:19.011011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:19.011179 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:19.011302 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:19.011557 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 04:34:19.011647 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:34:19.031675 20707 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:19.032707 20707 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:19.037824 20707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90fbb4cbc20a4b988204f69e69e0f016: Generated new TSK 0
I20260620 04:34:19.038561 20707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:19.046159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 20626
W20260620 04:34:19.050786 20611 connection.cc:535] client connection to 127.0.0.1:36637 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36637): Connection reset by peer (error 104)
2026-06-20T04:34:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:19 dist-test-slave-rzqm krb5kdc[20725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:19 dist-test-slave-rzqm krb5kdc[20725](info): set up 2 sockets
Jun 20 04:34:19 dist-test-slave-rzqm krb5kdc[20725](info): commencing operation
krb5kdc: starting...
W20260620 04:34:19.996867 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:20 dist-test-slave-rzqm krb5kdc[20725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:20.112213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:20.221241 20738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:20.221531 20738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:20.221591 20738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:20.225700 20738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:20.225782 20738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:20.225808 20738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:20.225837 20738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:20.225858 20738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:20.231532 20738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:20.232743 20738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:20.233852 20738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:20.242506 20743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:20.242492 20744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:20.242509 20746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:20.242687 20738 server_base.cc:1034] running on GCE node
I20260620 04:34:20.243363 20738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:20.244575 20738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:20.245738 20738 hybrid_clock.cc:648] HybridClock initialized: now 1781930060245706 us; error 48 us; skew 500 ppm
Jun 20 04:34:20 dist-test-slave-rzqm krb5kdc[20725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:20.252653 20738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:20.253693 20738 webserver.cc:460] Webserver started at http://127.0.0.1:40261/ using document root <none> and password file <none>
I20260620 04:34:20.254261 20738 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:20.254328 20738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:20.254532 20738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:20.256680 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "5afad22379ea4efaa0917df9fb5103cd"
format_stamp: "Formatted at 2026-06-20 04:34:20 on dist-test-slave-rzqm"
server_key: "3a3c4bce03eaa558cc54220a0c1fecd2"
server_key_iv: "0f0950060e09441c1afa9da44e7f9562"
server_key_version: "encryptionkey@0"
I20260620 04:34:20.257242 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "5afad22379ea4efaa0917df9fb5103cd"
format_stamp: "Formatted at 2026-06-20 04:34:20 on dist-test-slave-rzqm"
server_key: "3a3c4bce03eaa558cc54220a0c1fecd2"
server_key_iv: "0f0950060e09441c1afa9da44e7f9562"
server_key_version: "encryptionkey@0"
I20260620 04:34:20.261118 20738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 04:34:20.263682 20753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:20.264988 20738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:34:20.265162 20738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "5afad22379ea4efaa0917df9fb5103cd"
format_stamp: "Formatted at 2026-06-20 04:34:20 on dist-test-slave-rzqm"
server_key: "3a3c4bce03eaa558cc54220a0c1fecd2"
server_key_iv: "0f0950060e09441c1afa9da44e7f9562"
server_key_version: "encryptionkey@0"
I20260620 04:34:20.265271 20738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:20.278759 20738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:20.281865 20738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:20.282058 20738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:20.288838 20738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37469
I20260620 04:34:20.288858 20805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37469 every 8 connection(s)
I20260620 04:34:20.289853 20738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:20.292567 20806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:20.298058 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd: Bootstrap starting.
I20260620 04:34:20.298403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 20738
I20260620 04:34:20.298529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:20.298779 392 external_mini_cluster.cc:1428] Setting key 101661e429c08f72e67e08202635c6f8
I20260620 04:34:20.300490 20806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:20.301223 20806 log.cc:824] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:20.303048 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd: No bootstrap required, opened a new log
I20260620 04:34:20.307114 20806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afad22379ea4efaa0917df9fb5103cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } }
I20260620 04:34:20.307350 20806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:20.307408 20806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afad22379ea4efaa0917df9fb5103cd, State: Initialized, Role: FOLLOWER
I20260620 04:34:20.307603 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:20.300143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50257 (local address 127.0.0.1:37469)
0620 04:34:20.300537 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:20.300552 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:20.300587 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:34:20.301273 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:20.301290 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:20.301657 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:20.301931 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:20.302237 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:20.303199 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:20.304364 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:20.304837 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:20.304952 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:20.305854 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:20.305859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:20.305872 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:20.306275 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:20.306330 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:20.306352 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:20.306461 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:20.306634 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 04:34:20.306770 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260620 04:34:20.307869 20806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afad22379ea4efaa0917df9fb5103cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } }
I20260620 04:34:20.308008 20806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:20.308071 20806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:20.308166 20806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:20.309226 20806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afad22379ea4efaa0917df9fb5103cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } }
I20260620 04:34:20.309583 20806 leader_election.cc:302] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5afad22379ea4efaa0917df9fb5103cd; no voters:
I20260620 04:34:20.309935 20806 leader_election.cc:288] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:20.310043 20811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:20.310248 20811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [term 1 LEADER]: Becoming Leader. State: Replica: 5afad22379ea4efaa0917df9fb5103cd, State: Running, Role: LEADER
I20260620 04:34:20.310609 20811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afad22379ea4efaa0917df9fb5103cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } }
I20260620 04:34:20.310732 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:20.311246 20806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:20.312245 20813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5afad22379ea4efaa0917df9fb5103cd. Latest consensus state: current_term: 1 leader_uuid: "5afad22379ea4efaa0917df9fb5103cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afad22379ea4efaa0917df9fb5103cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } } }
I20260620 04:34:20.312247 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5afad22379ea4efaa0917df9fb5103cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afad22379ea4efaa0917df9fb5103cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } } }
I20260620 04:34:20.312388 20813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:20.312388 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd [sys.catalog]: This master's current role is: LEADER
W20260620 04:34:20.315970 20830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:20.316039 20830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:20.316138 20824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:20.317036 20824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:20.317972 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:20.312918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57437 (local address 127.0.0.1:37469)
0620 04:34:20.313044 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:20.313048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:20.313669 (+ 621us) server_negotiation.cc:407] Connection header received
0620 04:34:20.313773 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:20.313776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:20.313820 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:20.313895 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:20.314268 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:20.315015 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:20.316140 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:20.316447 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:20.316552 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:20.317482 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:20.317486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:20.317489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:20.317504 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:20.317530 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:20.317544 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:20.317637 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:20.317795 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 04:34:20.317830 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:34:20.322763 20824 catalog_manager.cc:1347] Generated new cluster ID: 6d3f5d4e0c024aac8c461f4ab710da3d
I20260620 04:34:20.322837 20824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:20.332288 20824 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:20.333334 20824 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:20.339823 20824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5afad22379ea4efaa0917df9fb5103cd: Generated new TSK 0
I20260620 04:34:20.340555 20824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:20.346428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 20738
W20260620 04:34:20.351253 20723 connection.cc:535] client connection to 127.0.0.1:37469 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37469): Connection reset by peer (error 104)
2026-06-20T04:34:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:20 dist-test-slave-rzqm krb5kdc[20837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:20 dist-test-slave-rzqm krb5kdc[20837](info): set up 2 sockets
Jun 20 04:34:20 dist-test-slave-rzqm krb5kdc[20837](info): commencing operation
krb5kdc: starting...
W20260620 04:34:21.300829 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:21 dist-test-slave-rzqm krb5kdc[20837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:21.414167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:21.524253 20850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:21.524549 20850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:21.524590 20850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:21.528684 20850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:21.528752 20850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:21.528770 20850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:21.528785 20850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:21.528800 20850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:21.534279 20850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:21.535408 20850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:21.536455 20850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:21.544852 20858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:21.544857 20856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:21.545116 20850 server_base.cc:1034] running on GCE node
W20260620 04:34:21.544852 20855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:21.545697 20850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:21.546811 20850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:21.547976 20850 hybrid_clock.cc:648] HybridClock initialized: now 1781930061547954 us; error 33 us; skew 500 ppm
Jun 20 04:34:21 dist-test-slave-rzqm krb5kdc[20837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:21.550624 20850 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:21.551466 20850 webserver.cc:460] Webserver started at http://127.0.0.1:35021/ using document root <none> and password file <none>
I20260620 04:34:21.551959 20850 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:21.552019 20850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:21.552187 20850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:21.554175 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "9ce14bb17bba4b26952f6bdc8db5914f"
format_stamp: "Formatted at 2026-06-20 04:34:21 on dist-test-slave-rzqm"
server_key: "a449b796d7dedf966f6e23c6cdb3e138"
server_key_iv: "a14c7ac066accca8c03fbb2ba3d0a204"
server_key_version: "encryptionkey@0"
I20260620 04:34:21.554666 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ce14bb17bba4b26952f6bdc8db5914f"
format_stamp: "Formatted at 2026-06-20 04:34:21 on dist-test-slave-rzqm"
server_key: "a449b796d7dedf966f6e23c6cdb3e138"
server_key_iv: "a14c7ac066accca8c03fbb2ba3d0a204"
server_key_version: "encryptionkey@0"
I20260620 04:34:21.558449 20850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:34:21.560748 20865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:21.562021 20850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:34:21.562122 20850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "9ce14bb17bba4b26952f6bdc8db5914f"
format_stamp: "Formatted at 2026-06-20 04:34:21 on dist-test-slave-rzqm"
server_key: "a449b796d7dedf966f6e23c6cdb3e138"
server_key_iv: "a14c7ac066accca8c03fbb2ba3d0a204"
server_key_version: "encryptionkey@0"
I20260620 04:34:21.562211 20850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:21.566562 20850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:21.569341 20850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:21.569480 20850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:21.576057 20850 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40901
I20260620 04:34:21.576097 20917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40901 every 8 connection(s)
I20260620 04:34:21.576958 20850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:21.579706 20918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:21.579950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 20850
I20260620 04:34:21.580082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:21.580377 392 external_mini_cluster.cc:1428] Setting key 8e639dbcfdf4f5bc454409ece799cb12
I20260620 04:34:21.585381 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:21.581700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39955 (local address 127.0.0.1:40901)
0620 04:34:21.582084 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:21.582100 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:21.582133 (+ 33us) server_negotiation.cc:407] Connection header received
0620 04:34:21.582786 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:21.582803 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:21.583093 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:21.583365 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:21.584145 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:21.584149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:21.584160 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:21.584170 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:21.584220 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:21.584238 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:21.584345 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:21.584435 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 04:34:21.584576 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":99,"threads_started":1}
I20260620 04:34:21.586181 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f: Bootstrap starting.
I20260620 04:34:21.587920 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:21.588466 20918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:21.589361 20918 log.cc:824] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:21.591565 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f: No bootstrap required, opened a new log
I20260620 04:34:21.591610 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:21.590180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37141 (local address 127.0.0.1:40901)
0620 04:34:21.590293 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:21.590297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:21.590448 (+ 151us) server_negotiation.cc:407] Connection header received
0620 04:34:21.590539 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:21.590542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:21.590574 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:21.590650 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:21.591283 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:21.591286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:21.591289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:21.591300 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:21.591328 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:21.591341 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:21.591434 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:21.591463 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 04:34:21.591498 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:34:21.595084 20918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce14bb17bba4b26952f6bdc8db5914f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260620 04:34:21.595310 20918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:21.595348 20918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ce14bb17bba4b26952f6bdc8db5914f, State: Initialized, Role: FOLLOWER
I20260620 04:34:21.595728 20918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce14bb17bba4b26952f6bdc8db5914f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260620 04:34:21.595851 20918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:21.595897 20918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:21.595968 20918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:21.596789 20918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce14bb17bba4b26952f6bdc8db5914f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260620 04:34:21.597110 20918 leader_election.cc:302] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ce14bb17bba4b26952f6bdc8db5914f; no voters:
I20260620 04:34:21.597368 20918 leader_election.cc:288] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:21.597473 20928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:21.597657 20928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [term 1 LEADER]: Becoming Leader. State: Replica: 9ce14bb17bba4b26952f6bdc8db5914f, State: Running, Role: LEADER
I20260620 04:34:21.597999 20928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce14bb17bba4b26952f6bdc8db5914f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } }
I20260620 04:34:21.598587 20918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:21.599535 20929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ce14bb17bba4b26952f6bdc8db5914f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce14bb17bba4b26952f6bdc8db5914f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } } }
I20260620 04:34:21.599665 20929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:21.600023 20937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:21.600003 20930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ce14bb17bba4b26952f6bdc8db5914f. Latest consensus state: current_term: 1 leader_uuid: "9ce14bb17bba4b26952f6bdc8db5914f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ce14bb17bba4b26952f6bdc8db5914f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40901 } } }
I20260620 04:34:21.600198 20930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:21.602780 20937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:21.608006 20937 catalog_manager.cc:1347] Generated new cluster ID: d58d9e9d45a4474cb06734e8b1e2832c
I20260620 04:34:21.608085 20937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:21.624840 20937 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:21.625185 20937 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:21.637565 20937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ce14bb17bba4b26952f6bdc8db5914f: Generated new TSK 0
I20260620 04:34:21.638286 20937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:21.641252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 20850
2026-06-20T04:34:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:21 dist-test-slave-rzqm krb5kdc[20949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:21 dist-test-slave-rzqm krb5kdc[20949](info): set up 2 sockets
Jun 20 04:34:21 dist-test-slave-rzqm krb5kdc[20949](info): commencing operation
krb5kdc: starting...
W20260620 04:34:22.596912 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:22 dist-test-slave-rzqm krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:22.712008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:22.818964 20962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:22.819237 20962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:22.819294 20962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:22.823247 20962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:22.823307 20962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:22.823333 20962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:22.823354 20962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:22.823374 20962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:22.828645 20962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:22.829682 20962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:22.830665 20962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:22.838871 20968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:22.839056 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:22.838987 20962 server_base.cc:1034] running on GCE node
W20260620 04:34:22.838893 20970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:22.839704 20962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:22.840781 20962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:22.841938 20962 hybrid_clock.cc:648] HybridClock initialized: now 1781930062841920 us; error 65 us; skew 500 ppm
Jun 20 04:34:22 dist-test-slave-rzqm krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:22.844537 20962 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:22.845438 20962 webserver.cc:460] Webserver started at http://127.0.0.1:45753/ using document root <none> and password file <none>
I20260620 04:34:22.845947 20962 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:22.846007 20962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:22.846169 20962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:22.848115 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6"
format_stamp: "Formatted at 2026-06-20 04:34:22 on dist-test-slave-rzqm"
server_key: "17c29963f8c5540fb8114cd663a6578d"
server_key_iv: "6ca284a3dfdc7da8a40624094aa60083"
server_key_version: "encryptionkey@0"
I20260620 04:34:22.848608 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6"
format_stamp: "Formatted at 2026-06-20 04:34:22 on dist-test-slave-rzqm"
server_key: "17c29963f8c5540fb8114cd663a6578d"
server_key_iv: "6ca284a3dfdc7da8a40624094aa60083"
server_key_version: "encryptionkey@0"
I20260620 04:34:22.852190 20962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:34:22.854558 20977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:22.855778 20962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:22.855876 20962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6"
format_stamp: "Formatted at 2026-06-20 04:34:22 on dist-test-slave-rzqm"
server_key: "17c29963f8c5540fb8114cd663a6578d"
server_key_iv: "6ca284a3dfdc7da8a40624094aa60083"
server_key_version: "encryptionkey@0"
I20260620 04:34:22.855963 20962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:22.868713 20962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:22.871869 20962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:22.872062 20962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:22.878856 20962 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34957
I20260620 04:34:22.878871 21029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34957 every 8 connection(s)
I20260620 04:34:22.879802 20962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:22.882579 21030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:22.887943 21030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6: Bootstrap starting.
I20260620 04:34:22.888058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 20962
I20260620 04:34:22.888186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:22.888471 392 external_mini_cluster.cc:1428] Setting key 3de8b349d2ef7e25923b66fc498c7da7
I20260620 04:34:22.890576 21030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:22.891304 21030 log.cc:824] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:22.893325 21030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6: No bootstrap required, opened a new log
Jun 20 04:34:22 dist-test-slave-rzqm krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:34:22.897470 21030 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } }
I20260620 04:34:22.897722 21030 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:22.897780 21030 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43eb93e1ed7429dbb69a4a1bee4e5e6, State: Initialized, Role: FOLLOWER
I20260620 04:34:22.898267 21030 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } }
I20260620 04:34:22.898427 21030 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:22.898497 21030 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:22.898597 21030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:22.899718 21030 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } }
I20260620 04:34:22.900117 21030 leader_election.cc:302] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43eb93e1ed7429dbb69a4a1bee4e5e6; no voters:
I20260620 04:34:22.900454 21030 leader_election.cc:288] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:22.900558 21035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:22.900769 21035 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [term 1 LEADER]: Becoming Leader. State: Replica: c43eb93e1ed7429dbb69a4a1bee4e5e6, State: Running, Role: LEADER
I20260620 04:34:22.901167 21035 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } }
I20260620 04:34:22.901809 21030 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:22.902819 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43eb93e1ed7429dbb69a4a1bee4e5e6. Latest consensus state: current_term: 1 leader_uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } } }
I20260620 04:34:22.902948 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:22.903229 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:22.889834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50991 (local address 127.0.0.1:34957)
0620 04:34:22.890240 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:22.890256 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:22.890291 (+ 35us) server_negotiation.cc:407] Connection header received
0620 04:34:22.890996 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:22.891015 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:22.891362 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:22.891688 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:22.892440 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:22.893563 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:22.894701 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:22.895233 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:22.895352 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:22.898061 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:22.898066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:22.898083 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:22.898115 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:22.900026 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:22.900453 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:22.900459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:22.900466 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:22.900618 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:22.900934 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:22.900937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:22.900939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:22.901445 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:22.901648 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:22.901913 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 04:34:22.902219 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260620 04:34:22.903231 21036 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43eb93e1ed7429dbb69a4a1bee4e5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } } }
I20260620 04:34:22.903338 21044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:22.903354 21036 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:22.906299 21044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:22.906605 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:22.911677 21044 catalog_manager.cc:1347] Generated new cluster ID: 22059a8c250b4ec3b1bf8075fa18e2f8
I20260620 04:34:22.911743 21044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:22.917327 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:22.908475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39597 (local address 127.0.0.1:34957)
0620 04:34:22.908587 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:22.908592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:22.908964 (+ 372us) server_negotiation.cc:407] Connection header received
0620 04:34:22.909104 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:22.909107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:22.909152 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:22.909229 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:22.910026 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:22.910796 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:22.911978 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:22.912321 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:22.912425 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:22.914100 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:22.914104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:22.914107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:22.914131 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:22.915696 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:22.916158 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:22.916161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:22.916162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:22.916233 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:22.916486 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:22.916489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:22.916490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:22.916677 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:22.916779 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:22.917053 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 04:34:22.917167 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:34:22.928509 21044 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:22.929883 21044 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:22.937727 21044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43eb93e1ed7429dbb69a4a1bee4e5e6: Generated new TSK 0
I20260620 04:34:22.938539 21044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:22.948776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 20962
2026-06-20T04:34:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:23 dist-test-slave-rzqm krb5kdc[21061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:23 dist-test-slave-rzqm krb5kdc[21061](info): set up 2 sockets
Jun 20 04:34:23 dist-test-slave-rzqm krb5kdc[21061](info): commencing operation
krb5kdc: starting...
W20260620 04:34:23.908984 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:23 dist-test-slave-rzqm krb5kdc[21061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:24.023907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:24.130398 21074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:24.130687 21074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:24.130746 21074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:24.134801 21074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:24.134876 21074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:24.134903 21074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:24.134923 21074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:24.134944 21074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:24.140407 21074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:24.141566 21074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:24.142613 21074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:24.150900 21080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:24.150921 21079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:24.150900 21082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:24.151172 21074 server_base.cc:1034] running on GCE node
I20260620 04:34:24.151773 21074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:24.152846 21074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:24.154049 21074 hybrid_clock.cc:648] HybridClock initialized: now 1781930064154018 us; error 42 us; skew 500 ppm
Jun 20 04:34:24 dist-test-slave-rzqm krb5kdc[21061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:24.156898 21074 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:24.157793 21074 webserver.cc:460] Webserver started at http://127.0.0.1:43251/ using document root <none> and password file <none>
I20260620 04:34:24.158324 21074 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:24.158388 21074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:24.158589 21074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:24.160614 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "4d1c9b389bc34e60be27f0e4ae182292"
format_stamp: "Formatted at 2026-06-20 04:34:24 on dist-test-slave-rzqm"
server_key: "bbee809d05cddf85d5436af1d36d9d83"
server_key_iv: "dd6d0e0697bc77fbab9345f732d65258"
server_key_version: "encryptionkey@0"
I20260620 04:34:24.161175 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d1c9b389bc34e60be27f0e4ae182292"
format_stamp: "Formatted at 2026-06-20 04:34:24 on dist-test-slave-rzqm"
server_key: "bbee809d05cddf85d5436af1d36d9d83"
server_key_iv: "dd6d0e0697bc77fbab9345f732d65258"
server_key_version: "encryptionkey@0"
I20260620 04:34:24.164709 21074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:24.167163 21089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:24.168509 21074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:34:24.168632 21074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "4d1c9b389bc34e60be27f0e4ae182292"
format_stamp: "Formatted at 2026-06-20 04:34:24 on dist-test-slave-rzqm"
server_key: "bbee809d05cddf85d5436af1d36d9d83"
server_key_iv: "dd6d0e0697bc77fbab9345f732d65258"
server_key_version: "encryptionkey@0"
I20260620 04:34:24.168740 21074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:24.179194 21074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:24.182260 21074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:24.182440 21074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:24.188987 21074 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40745
I20260620 04:34:24.189075 21141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40745 every 8 connection(s)
I20260620 04:34:24.189976 21074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:24.192750 21142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:24.198352 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292: Bootstrap starting.
I20260620 04:34:24.199905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 21074
I20260620 04:34:24.200048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:24.200340 392 external_mini_cluster.cc:1428] Setting key 91c4aab72fe7f5afff6940dbf947b7a9
I20260620 04:34:24.201179 21142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:24.202160 21142 log.cc:824] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:24.204380 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292: No bootstrap required, opened a new log
I20260620 04:34:24.208549 21142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1c9b389bc34e60be27f0e4ae182292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260620 04:34:24.208854 21142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:24.208916 21142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1c9b389bc34e60be27f0e4ae182292, State: Initialized, Role: FOLLOWER
I20260620 04:34:24.209414 21142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1c9b389bc34e60be27f0e4ae182292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260620 04:34:24.209555 21142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:24.209640 21142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:24.209743 21142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:24.210152 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:24.201516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45983 (local address 127.0.0.1:40745)
0620 04:34:24.201941 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:24.201959 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:24.202001 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:34:24.202791 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:24.202812 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:24.203215 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:24.203574 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:24.203975 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:24.205409 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:24.206345 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:24.207101 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:24.207275 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:24.208059 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:24.208065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:24.208083 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:24.208587 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:24.208659 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:24.208690 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:24.208785 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:24.208957 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 04:34:24.209208 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260620 04:34:24.210822 21142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1c9b389bc34e60be27f0e4ae182292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260620 04:34:24.211205 21142 leader_election.cc:302] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d1c9b389bc34e60be27f0e4ae182292; no voters:
I20260620 04:34:24.211547 21142 leader_election.cc:288] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:24.211642 21147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:24.211809 21147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [term 1 LEADER]: Becoming Leader. State: Replica: 4d1c9b389bc34e60be27f0e4ae182292, State: Running, Role: LEADER
I20260620 04:34:24.212458 21147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1c9b389bc34e60be27f0e4ae182292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260620 04:34:24.212867 21142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:24.213160 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:24.214100 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d1c9b389bc34e60be27f0e4ae182292. Latest consensus state: current_term: 1 leader_uuid: "4d1c9b389bc34e60be27f0e4ae182292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1c9b389bc34e60be27f0e4ae182292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } } }
I20260620 04:34:24.214241 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:24.214569 21148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d1c9b389bc34e60be27f0e4ae182292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1c9b389bc34e60be27f0e4ae182292" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } } }
I20260620 04:34:24.214651 21148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292 [sys.catalog]: This master's current role is: LEADER
W20260620 04:34:24.217758 21166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:24.217841 21166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:24.217897 21159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:24.218729 21159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:24.219534 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:24.215555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52971 (local address 127.0.0.1:40745)
0620 04:34:24.215668 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:24.215672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:24.215957 (+ 285us) server_negotiation.cc:407] Connection header received
0620 04:34:24.216143 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:24.216146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:24.216190 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:24.216238 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:24.216630 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:24.217274 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:24.218106 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:24.218340 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:24.218426 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:24.219157 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:24.219160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:24.219163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:24.219175 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:24.219201 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:24.219211 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:24.219269 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:24.219352 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 04:34:24.219401 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:34:24.224817 21159 catalog_manager.cc:1347] Generated new cluster ID: 2919100130534021b73265d8885dfc2e
I20260620 04:34:24.224879 21159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:24.238162 21159 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:24.239068 21159 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:24.246121 21159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d1c9b389bc34e60be27f0e4ae182292: Generated new TSK 0
I20260620 04:34:24.246848 21159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:24.252733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 21074
2026-06-20T04:34:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:24 dist-test-slave-rzqm krb5kdc[21173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:24 dist-test-slave-rzqm krb5kdc[21173](info): set up 2 sockets
Jun 20 04:34:24 dist-test-slave-rzqm krb5kdc[21173](info): commencing operation
krb5kdc: starting...
W20260620 04:34:25.196949 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:25 dist-test-slave-rzqm krb5kdc[21173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:25.312206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:25.419735 21186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:25.420004 21186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:25.420046 21186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:25.424086 21186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:25.424158 21186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:25.424177 21186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:25.424192 21186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:25.424206 21186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:25.429702 21186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:25.430790 21186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:25.431834 21186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:25.440651 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:25.440722 21186 server_base.cc:1034] running on GCE node
W20260620 04:34:25.440711 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:25.440651 21194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:25.441495 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:25.442646 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:25.443820 21186 hybrid_clock.cc:648] HybridClock initialized: now 1781930065443795 us; error 37 us; skew 500 ppm
Jun 20 04:34:25 dist-test-slave-rzqm krb5kdc[21173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:25.451318 21186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:25.452427 21186 webserver.cc:460] Webserver started at http://127.0.0.1:38225/ using document root <none> and password file <none>
I20260620 04:34:25.453109 21186 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:25.453190 21186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:25.453421 21186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:25.455729 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "ae1cf284ed324e4f8522101e9c57212a"
format_stamp: "Formatted at 2026-06-20 04:34:25 on dist-test-slave-rzqm"
server_key: "a644cad8bfd866f7ab102ae943546aab"
server_key_iv: "bb711eea1412586f7e4589dea50f706b"
server_key_version: "encryptionkey@0"
I20260620 04:34:25.456357 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae1cf284ed324e4f8522101e9c57212a"
format_stamp: "Formatted at 2026-06-20 04:34:25 on dist-test-slave-rzqm"
server_key: "a644cad8bfd866f7ab102ae943546aab"
server_key_iv: "bb711eea1412586f7e4589dea50f706b"
server_key_version: "encryptionkey@0"
I20260620 04:34:25.460573 21186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:34:25.463253 21201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:25.464526 21186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:34:25.464651 21186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "ae1cf284ed324e4f8522101e9c57212a"
format_stamp: "Formatted at 2026-06-20 04:34:25 on dist-test-slave-rzqm"
server_key: "a644cad8bfd866f7ab102ae943546aab"
server_key_iv: "bb711eea1412586f7e4589dea50f706b"
server_key_version: "encryptionkey@0"
I20260620 04:34:25.464764 21186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:25.469447 21186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:25.472374 21186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:25.472539 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:25.479354 21186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42371
I20260620 04:34:25.479369 21253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42371 every 8 connection(s)
I20260620 04:34:25.480307 21186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:25.483047 21254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:25.487814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 21186
I20260620 04:34:25.487958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:25.488255 392 external_mini_cluster.cc:1428] Setting key 8c6ee0f295f24cdd813a00c3697e4081
I20260620 04:34:25.488679 21254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a: Bootstrap starting.
I20260620 04:34:25.491195 21254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:25.491976 21254 log.cc:824] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:25.493678 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:25.489713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45975 (local address 127.0.0.1:42371)
0620 04:34:25.490194 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:25.490209 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:25.490247 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:34:25.490910 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:25.490929 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:25.491231 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:25.491630 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:25.492406 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:25.492412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:25.492426 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:25.492436 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:25.492492 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:25.492512 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:25.492613 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:25.492733 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 04:34:25.492902 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":143,"threads_started":1}
I20260620 04:34:25.494050 21254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a: No bootstrap required, opened a new log
I20260620 04:34:25.496631 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:25.498095 21254 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1cf284ed324e4f8522101e9c57212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42371 } }
I20260620 04:34:25.498327 21254 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:25.498381 21254 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1cf284ed324e4f8522101e9c57212a, State: Initialized, Role: FOLLOWER
I20260620 04:34:25.498847 21254 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1cf284ed324e4f8522101e9c57212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42371 } }
I20260620 04:34:25.499006 21254 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:25.499073 21254 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:25.499168 21254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:25.500178 21254 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1cf284ed324e4f8522101e9c57212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42371 } }
I20260620 04:34:25.500316 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:25.498616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36691 (local address 127.0.0.1:42371)
0620 04:34:25.498720 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:25.498724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:25.498879 (+ 155us) server_negotiation.cc:407] Connection header received
0620 04:34:25.499048 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:25.499051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:25.499088 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:25.499152 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:25.499860 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:25.499864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:25.499867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:25.499878 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:25.499908 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:25.499921 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:25.500003 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:25.500106 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 04:34:25.500145 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 04:34:25.500579 21254 leader_election.cc:302] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae1cf284ed324e4f8522101e9c57212a; no voters:
I20260620 04:34:25.500903 21254 leader_election.cc:288] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:25.501051 21264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:25.501257 21264 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [term 1 LEADER]: Becoming Leader. State: Replica: ae1cf284ed324e4f8522101e9c57212a, State: Running, Role: LEADER
I20260620 04:34:25.501613 21264 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1cf284ed324e4f8522101e9c57212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42371 } }
I20260620 04:34:25.502131 21254 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:25.503221 21266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae1cf284ed324e4f8522101e9c57212a. Latest consensus state: current_term: 1 leader_uuid: "ae1cf284ed324e4f8522101e9c57212a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1cf284ed324e4f8522101e9c57212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42371 } } }
I20260620 04:34:25.503212 21265 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae1cf284ed324e4f8522101e9c57212a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1cf284ed324e4f8522101e9c57212a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42371 } } }
I20260620 04:34:25.503347 21265 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:25.503347 21266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:25.503750 21274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:25.507028 21274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:25.512753 21274 catalog_manager.cc:1347] Generated new cluster ID: 88fb4c9034f8436a839dd1d122539117
I20260620 04:34:25.512835 21274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:25.537935 21274 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:25.538285 21274 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:25.554709 21274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae1cf284ed324e4f8522101e9c57212a: Generated new TSK 0
I20260620 04:34:25.555436 21274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:25.560778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 21186
2026-06-20T04:34:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:25 dist-test-slave-rzqm krb5kdc[21285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:25 dist-test-slave-rzqm krb5kdc[21285](info): set up 2 sockets
Jun 20 04:34:25 dist-test-slave-rzqm krb5kdc[21285](info): commencing operation
krb5kdc: starting...
W20260620 04:34:26.509119 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:26 dist-test-slave-rzqm krb5kdc[21285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:26.625949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:26.732189 21298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:26.732488 21298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:26.732532 21298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:26.736575 21298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:26.736637 21298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:26.736655 21298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:26.736671 21298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:26.736685 21298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:26.741972 21298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:26.742952 21298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:26.744048 21298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:26.752543 21303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:26.752554 21304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:26.752543 21306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:26.752826 21298 server_base.cc:1034] running on GCE node
I20260620 04:34:26.753435 21298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:26.754580 21298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:26.755752 21298 hybrid_clock.cc:648] HybridClock initialized: now 1781930066755727 us; error 38 us; skew 500 ppm
Jun 20 04:34:26 dist-test-slave-rzqm krb5kdc[21285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:26.758512 21298 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:26.759418 21298 webserver.cc:460] Webserver started at http://127.0.0.1:34209/ using document root <none> and password file <none>
I20260620 04:34:26.759964 21298 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:26.760025 21298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:26.760223 21298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:26.762367 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "aa2c3518b4604b97b82bbb43919b52a9"
format_stamp: "Formatted at 2026-06-20 04:34:26 on dist-test-slave-rzqm"
server_key: "0a8ee2595e8e8161c5b1f5b93c5155f3"
server_key_iv: "75cb27660d1cd02e4992646238c96270"
server_key_version: "encryptionkey@0"
I20260620 04:34:26.762907 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa2c3518b4604b97b82bbb43919b52a9"
format_stamp: "Formatted at 2026-06-20 04:34:26 on dist-test-slave-rzqm"
server_key: "0a8ee2595e8e8161c5b1f5b93c5155f3"
server_key_iv: "75cb27660d1cd02e4992646238c96270"
server_key_version: "encryptionkey@0"
I20260620 04:34:26.766696 21298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:34:26.769215 21313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:26.770491 21298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 04:34:26.770612 21298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "aa2c3518b4604b97b82bbb43919b52a9"
format_stamp: "Formatted at 2026-06-20 04:34:26 on dist-test-slave-rzqm"
server_key: "0a8ee2595e8e8161c5b1f5b93c5155f3"
server_key_iv: "75cb27660d1cd02e4992646238c96270"
server_key_version: "encryptionkey@0"
I20260620 04:34:26.770709 21298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:26.791749 21298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:26.794847 21298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:26.795025 21298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:26.801523 21298 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35003
I20260620 04:34:26.801620 21365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35003 every 8 connection(s)
I20260620 04:34:26.802452 21298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:26.805157 21366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:26.810690 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9: Bootstrap starting.
I20260620 04:34:26.811806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 21298
I20260620 04:34:26.811962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:26.812282 392 external_mini_cluster.cc:1428] Setting key 20a4c87374a4ab4bef9bdf93167b7fd9
I20260620 04:34:26.813496 21366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:26.814335 21366 log.cc:824] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:26.816097 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9: No bootstrap required, opened a new log
I20260620 04:34:26.819609 21366 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2c3518b4604b97b82bbb43919b52a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } }
I20260620 04:34:26.819842 21366 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:26.819914 21366 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa2c3518b4604b97b82bbb43919b52a9, State: Initialized, Role: FOLLOWER
I20260620 04:34:26.820407 21366 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2c3518b4604b97b82bbb43919b52a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } }
I20260620 04:34:26.820573 21366 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Jun 20 04:34:26 dist-test-slave-rzqm krb5kdc[21285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:34:26.820636 21366 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:26.820725 21366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:26.821674 21366 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2c3518b4604b97b82bbb43919b52a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } }
I20260620 04:34:26.822037 21366 leader_election.cc:302] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa2c3518b4604b97b82bbb43919b52a9; no voters:
I20260620 04:34:26.822369 21366 leader_election.cc:288] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:26.822453 21371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:26.822636 21371 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [term 1 LEADER]: Becoming Leader. State: Replica: aa2c3518b4604b97b82bbb43919b52a9, State: Running, Role: LEADER
I20260620 04:34:26.823048 21371 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2c3518b4604b97b82bbb43919b52a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } }
I20260620 04:34:26.823721 21366 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:26.825580 21372 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa2c3518b4604b97b82bbb43919b52a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2c3518b4604b97b82bbb43919b52a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } } }
I20260620 04:34:26.825704 21372 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:26.826052 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa2c3518b4604b97b82bbb43919b52a9. Latest consensus state: current_term: 1 leader_uuid: "aa2c3518b4604b97b82bbb43919b52a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2c3518b4604b97b82bbb43919b52a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } } }
I20260620 04:34:26.826277 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:26.826905 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:26.813480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:58813 (local address 127.0.0.1:35003)
0620 04:34:26.813948 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:26.813964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:26.813998 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:34:26.814685 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:26.814704 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:26.815040 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:26.815351 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:26.816133 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:26.817150 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:26.817968 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:26.818439 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:26.818599 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:26.821340 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:26.821345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:26.821359 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:26.821387 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:26.823255 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:26.823780 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:26.823786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:26.823792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:26.823929 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:26.824290 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:26.824293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:26.824295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:26.824796 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:26.825033 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:26.825691 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 04:34:26.825927 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":181,"threads_started":1}
W20260620 04:34:26.829362 21385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:26.829481 21385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:26.829563 21386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:26.829739 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:26.830410 21386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:26.836298 21386 catalog_manager.cc:1347] Generated new cluster ID: 5f631250836549abb6333d831ba8bcbb
I20260620 04:34:26.836364 21386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:26.840265 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:26.831525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:56403 (local address 127.0.0.1:35003)
0620 04:34:26.831629 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:26.831633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:26.831904 (+ 271us) server_negotiation.cc:407] Connection header received
0620 04:34:26.832001 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:26.832004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:26.832048 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:26.832113 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:26.832923 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:26.834113 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:26.835044 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:26.835299 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:26.835384 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:26.836884 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:26.836888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:26.836891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:26.836917 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:26.838557 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:26.839060 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:26.839064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:26.839066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:26.839152 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:26.839439 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:26.839442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:26.839444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:26.839651 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:26.839806 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:26.840018 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 04:34:26.840115 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 04:34:26.853211 21386 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:26.854456 21386 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:26.863336 21386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa2c3518b4604b97b82bbb43919b52a9: Generated new TSK 0
I20260620 04:34:26.864147 21386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:26.875399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 21298
W20260620 04:34:26.879989 21283 connection.cc:535] client connection to 127.0.0.1:35003 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35003): Connection reset by peer (error 104)
2026-06-20T04:34:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:26 dist-test-slave-rzqm krb5kdc[21397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:26 dist-test-slave-rzqm krb5kdc[21397](info): set up 2 sockets
Jun 20 04:34:26 dist-test-slave-rzqm krb5kdc[21397](info): commencing operation
krb5kdc: starting...
W20260620 04:34:27.828900 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:27 dist-test-slave-rzqm krb5kdc[21397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:27.944270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:28.053594 21410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:28.053884 21410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:28.053946 21410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:28.057991 21410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:28.058061 21410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:28.058089 21410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:28.058110 21410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:28.058138 21410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:28.063637 21410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:28.064730 21410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:28.065790 21410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:28.073889 21418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:28.073908 21415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:28.074024 21416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:28.074359 21410 server_base.cc:1034] running on GCE node
I20260620 04:34:28.074963 21410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:28.076086 21410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:28.077252 21410 hybrid_clock.cc:648] HybridClock initialized: now 1781930068077236 us; error 28 us; skew 500 ppm
Jun 20 04:34:28 dist-test-slave-rzqm krb5kdc[21397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:28.079883 21410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:28.080772 21410 webserver.cc:460] Webserver started at http://127.0.0.1:38785/ using document root <none> and password file <none>
I20260620 04:34:28.081343 21410 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:28.081435 21410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:28.081643 21410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:28.083683 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "eced2a28e3594f3abc99a138ccfebf4a"
format_stamp: "Formatted at 2026-06-20 04:34:28 on dist-test-slave-rzqm"
server_key: "ec9b60f83acf5c634a22eacd7ff2d6f5"
server_key_iv: "50c47954bb33dcb32340d52c7e4c43e5"
server_key_version: "encryptionkey@0"
I20260620 04:34:28.084213 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "eced2a28e3594f3abc99a138ccfebf4a"
format_stamp: "Formatted at 2026-06-20 04:34:28 on dist-test-slave-rzqm"
server_key: "ec9b60f83acf5c634a22eacd7ff2d6f5"
server_key_iv: "50c47954bb33dcb32340d52c7e4c43e5"
server_key_version: "encryptionkey@0"
I20260620 04:34:28.087934 21410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:34:28.090466 21425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:28.091744 21410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:28.091866 21410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "eced2a28e3594f3abc99a138ccfebf4a"
format_stamp: "Formatted at 2026-06-20 04:34:28 on dist-test-slave-rzqm"
server_key: "ec9b60f83acf5c634a22eacd7ff2d6f5"
server_key_iv: "50c47954bb33dcb32340d52c7e4c43e5"
server_key_version: "encryptionkey@0"
I20260620 04:34:28.091969 21410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:28.112869 21410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:28.116217 21410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:28.116410 21410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:28.123586 21410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38843
I20260620 04:34:28.123648 21477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38843 every 8 connection(s)
I20260620 04:34:28.124636 21410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:28.127660 21478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:28.130416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 21410
I20260620 04:34:28.130620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:28.130937 392 external_mini_cluster.cc:1428] Setting key c6b14ad210e576496008c0e755d8fcdf
I20260620 04:34:28.134298 21478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a: Bootstrap starting.
I20260620 04:34:28.137293 21478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:28.138171 21478 log.cc:824] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:28.140165 21478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a: No bootstrap required, opened a new log
I20260620 04:34:28.141425 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:28.132423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:59841 (local address 127.0.0.1:38843)
0620 04:34:28.132908 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:28.132928 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:28.132976 (+ 48us) server_negotiation.cc:407] Connection header received
0620 04:34:28.133841 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:28.133866 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:28.134331 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:28.134757 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:28.135185 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:28.136801 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:28.137854 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:28.138595 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:28.138760 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:28.139653 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:28.139658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:28.139674 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:28.140093 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:28.140148 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:28.140172 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:28.140290 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:28.140491 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 04:34:28.140628 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":132,"threads_started":1}
I20260620 04:34:28.143658 21478 raft_consensus.cc:348] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced2a28e3594f3abc99a138ccfebf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38843 } }
I20260620 04:34:28.143932 21478 raft_consensus.cc:374] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:28.143992 21478 raft_consensus.cc:729] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eced2a28e3594f3abc99a138ccfebf4a, State: Initialized, Role: FOLLOWER
I20260620 04:34:28.144452 21478 consensus_queue.cc:260] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced2a28e3594f3abc99a138ccfebf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38843 } }
I20260620 04:34:28.144594 21478 raft_consensus.cc:388] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:28.144708 21478 raft_consensus.cc:482] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:28.144816 21478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:28.144830 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:28.145774 21478 raft_consensus.cc:504] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced2a28e3594f3abc99a138ccfebf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38843 } }
I20260620 04:34:28.146112 21478 leader_election.cc:302] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eced2a28e3594f3abc99a138ccfebf4a; no voters:
I20260620 04:34:28.146416 21478 leader_election.cc:288] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:28.146493 21487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:28.146695 21487 raft_consensus.cc:686] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [term 1 LEADER]: Becoming Leader. State: Replica: eced2a28e3594f3abc99a138ccfebf4a, State: Running, Role: LEADER
I20260620 04:34:28.147147 21487 consensus_queue.cc:237] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced2a28e3594f3abc99a138ccfebf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38843 } }
I20260620 04:34:28.147780 21478 sys_catalog.cc:564] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:28.148346 21487 sys_catalog.cc:455] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eced2a28e3594f3abc99a138ccfebf4a. Latest consensus state: current_term: 1 leader_uuid: "eced2a28e3594f3abc99a138ccfebf4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced2a28e3594f3abc99a138ccfebf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38843 } } }
I20260620 04:34:28.148521 21487 sys_catalog.cc:458] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:28.148754 21487 sys_catalog.cc:455] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eced2a28e3594f3abc99a138ccfebf4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eced2a28e3594f3abc99a138ccfebf4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38843 } } }
I20260620 04:34:28.148878 21487 sys_catalog.cc:458] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:28.149391 21496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:28.151459 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:28.146780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:45993 (local address 127.0.0.1:38843)
0620 04:34:28.146889 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:28.146892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:28.147115 (+ 223us) server_negotiation.cc:407] Connection header received
0620 04:34:28.147222 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:28.147225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:28.147276 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:28.147345 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:28.147605 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:28.148193 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:28.149290 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:28.149602 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:28.149677 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:28.150708 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:28.150712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:28.150715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:28.150731 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:28.150758 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:28.150771 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:28.150849 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:28.151293 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 04:34:28.151330 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:34:28.152719 21496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:28.158346 21496 catalog_manager.cc:1347] Generated new cluster ID: cee067104e1045c792ab9744fe17b9ac
I20260620 04:34:28.158424 21496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:28.187282 21496 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:28.188253 21496 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:28.197597 21496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eced2a28e3594f3abc99a138ccfebf4a: Generated new TSK 0
I20260620 04:34:28.198339 21496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:28.202405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 21410
W20260620 04:34:28.206965 21395 connection.cc:535] client connection to 127.0.0.1:38843 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38843): Connection reset by peer (error 104)
2026-06-20T04:34:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:28 dist-test-slave-rzqm krb5kdc[21509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:28 dist-test-slave-rzqm krb5kdc[21509](info): set up 2 sockets
Jun 20 04:34:28 dist-test-slave-rzqm krb5kdc[21509](info): commencing operation
krb5kdc: starting...
W20260620 04:34:29.161273 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:29 dist-test-slave-rzqm krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:29.279395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:29.389549 21522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:29.389868 21522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:29.389932 21522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:29.394285 21522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:29.394348 21522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:29.394366 21522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:29.394383 21522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:29.394411 21522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:29.399839 21522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:29.400894 21522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:29.401943 21522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:29.410229 21527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:29.410202 21530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:29.410218 21528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:29.410362 21522 server_base.cc:1034] running on GCE node
I20260620 04:34:29.411136 21522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:29.412214 21522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:29.413393 21522 hybrid_clock.cc:648] HybridClock initialized: now 1781930069413364 us; error 39 us; skew 500 ppm
Jun 20 04:34:29 dist-test-slave-rzqm krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:29.416237 21522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:29.417192 21522 webserver.cc:460] Webserver started at http://127.0.0.1:46589/ using document root <none> and password file <none>
I20260620 04:34:29.417737 21522 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:29.417800 21522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:29.417992 21522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:29.419994 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "081b23089fde4bd889a958e948d38b00"
format_stamp: "Formatted at 2026-06-20 04:34:29 on dist-test-slave-rzqm"
server_key: "d1cfef431898e23f01ab45295e33bde1"
server_key_iv: "5956c44e51dc71d305adf586a6f65c8c"
server_key_version: "encryptionkey@0"
I20260620 04:34:29.420543 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "081b23089fde4bd889a958e948d38b00"
format_stamp: "Formatted at 2026-06-20 04:34:29 on dist-test-slave-rzqm"
server_key: "d1cfef431898e23f01ab45295e33bde1"
server_key_iv: "5956c44e51dc71d305adf586a6f65c8c"
server_key_version: "encryptionkey@0"
I20260620 04:34:29.424388 21522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:34:29.426857 21537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:29.428076 21522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 04:34:29.428188 21522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "081b23089fde4bd889a958e948d38b00"
format_stamp: "Formatted at 2026-06-20 04:34:29 on dist-test-slave-rzqm"
server_key: "d1cfef431898e23f01ab45295e33bde1"
server_key_iv: "5956c44e51dc71d305adf586a6f65c8c"
server_key_version: "encryptionkey@0"
I20260620 04:34:29.428287 21522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:29.432763 21522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:29.435897 21522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:29.436142 21522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:29.443029 21522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33867
I20260620 04:34:29.443049 21589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33867 every 8 connection(s)
I20260620 04:34:29.443948 21522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:29.445394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 21522
I20260620 04:34:29.445552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:29.445798 392 external_mini_cluster.cc:1428] Setting key fbe5c56932b2c8152b816f03741997cb
I20260620 04:34:29.446934 21590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:29.450779 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:29.447060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:59111 (local address 127.0.0.1:33867)
0620 04:34:29.447456 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:29.447473 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:29.447507 (+ 34us) server_negotiation.cc:407] Connection header received
0620 04:34:29.448137 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:29.448153 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:29.448432 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:29.448712 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:29.449582 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:29.449587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:29.449599 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:29.449610 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:29.449667 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:29.449686 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:29.449802 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:29.449891 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 04:34:29.450044 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260620 04:34:29.453259 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00: Bootstrap starting.
I20260620 04:34:29.453917 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:29.455600 21590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:29.456421 21590 log.cc:824] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:29.457679 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:29.455936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:39663 (local address 127.0.0.1:33867)
0620 04:34:29.456067 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:29.456071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:29.456236 (+ 165us) server_negotiation.cc:407] Connection header received
0620 04:34:29.456335 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:29.456338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:29.456378 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:29.456462 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:29.457296 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:29.457299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:29.457301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:29.457311 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:29.457337 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:29.457347 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:29.457420 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:29.457525 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 04:34:29.457560 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:34:29.458561 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00: No bootstrap required, opened a new log
I20260620 04:34:29.462815 21590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b23089fde4bd889a958e948d38b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260620 04:34:29.463068 21590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:29.463120 21590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081b23089fde4bd889a958e948d38b00, State: Initialized, Role: FOLLOWER
I20260620 04:34:29.463536 21590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b23089fde4bd889a958e948d38b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260620 04:34:29.463670 21590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:29.463732 21590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:29.463826 21590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:29.464735 21590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b23089fde4bd889a958e948d38b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260620 04:34:29.465116 21590 leader_election.cc:302] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 081b23089fde4bd889a958e948d38b00; no voters:
I20260620 04:34:29.465420 21590 leader_election.cc:288] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:29.465523 21600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:29.465742 21600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [term 1 LEADER]: Becoming Leader. State: Replica: 081b23089fde4bd889a958e948d38b00, State: Running, Role: LEADER
I20260620 04:34:29.466094 21600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b23089fde4bd889a958e948d38b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } }
I20260620 04:34:29.466596 21590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:29.467638 21602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 081b23089fde4bd889a958e948d38b00. Latest consensus state: current_term: 1 leader_uuid: "081b23089fde4bd889a958e948d38b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b23089fde4bd889a958e948d38b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } } }
I20260620 04:34:29.467753 21602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:29.468127 21609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:29.468098 21601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "081b23089fde4bd889a958e948d38b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "081b23089fde4bd889a958e948d38b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33867 } } }
I20260620 04:34:29.468241 21601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:29.471421 21609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:29.476523 21609 catalog_manager.cc:1347] Generated new cluster ID: 278df063b7234ae88654c96db21bc5ec
I20260620 04:34:29.476606 21609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:29.486661 21609 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:29.487114 21609 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:29.503722 21609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 081b23089fde4bd889a958e948d38b00: Generated new TSK 0
I20260620 04:34:29.504642 21609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:29.516254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 21522
2026-06-20T04:34:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:29 dist-test-slave-rzqm krb5kdc[21621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:29 dist-test-slave-rzqm krb5kdc[21621](info): set up 2 sockets
Jun 20 04:34:29 dist-test-slave-rzqm krb5kdc[21621](info): commencing operation
krb5kdc: starting...
W20260620 04:34:30.476984 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:30 dist-test-slave-rzqm krb5kdc[21621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:30.594295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:30.702227 21634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:30.702502 21634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:30.702560 21634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:30.706566 21634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:30.706637 21634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:30.706663 21634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:30.706682 21634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:30.706704 21634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:30.712008 21634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:30.713052 21634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:30.714038 21634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:30.722404 21639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:30.722406 21642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:30.722422 21640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:30.722486 21634 server_base.cc:1034] running on GCE node
I20260620 04:34:30.723305 21634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:30.724723 21634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:30.725906 21634 hybrid_clock.cc:648] HybridClock initialized: now 1781930070725846 us; error 81 us; skew 500 ppm
Jun 20 04:34:30 dist-test-slave-rzqm krb5kdc[21621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:30.733183 21634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:30.734138 21634 webserver.cc:460] Webserver started at http://127.0.0.1:44601/ using document root <none> and password file <none>
I20260620 04:34:30.734719 21634 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:30.734789 21634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:30.734994 21634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:30.737118 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "da4216b1509b4913890b2a5c65b5c6b0"
format_stamp: "Formatted at 2026-06-20 04:34:30 on dist-test-slave-rzqm"
server_key: "0f8ac803f84a8be733bfdcb936073345"
server_key_iv: "3604558461a8a72ee281147ea2e2e8cc"
server_key_version: "encryptionkey@0"
I20260620 04:34:30.737686 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "da4216b1509b4913890b2a5c65b5c6b0"
format_stamp: "Formatted at 2026-06-20 04:34:30 on dist-test-slave-rzqm"
server_key: "0f8ac803f84a8be733bfdcb936073345"
server_key_iv: "3604558461a8a72ee281147ea2e2e8cc"
server_key_version: "encryptionkey@0"
I20260620 04:34:30.741381 21634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:34:30.743780 21649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:30.745080 21634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:30.745203 21634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "da4216b1509b4913890b2a5c65b5c6b0"
format_stamp: "Formatted at 2026-06-20 04:34:30 on dist-test-slave-rzqm"
server_key: "0f8ac803f84a8be733bfdcb936073345"
server_key_iv: "3604558461a8a72ee281147ea2e2e8cc"
server_key_version: "encryptionkey@0"
I20260620 04:34:30.745302 21634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:30.758641 21634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:30.761883 21634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:30.762073 21634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:30.769122 21634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41355
I20260620 04:34:30.769155 21701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41355 every 8 connection(s)
I20260620 04:34:30.770246 21634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:30.773134 21702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:30.778642 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0: Bootstrap starting.
I20260620 04:34:30.780306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 21634
I20260620 04:34:30.780458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:30.780715 392 external_mini_cluster.cc:1428] Setting key 25a0e229d260a1cd1995f6931c2d196f
I20260620 04:34:30.781253 21702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:30.782125 21702 log.cc:824] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:30.784328 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0: No bootstrap required, opened a new log
I20260620 04:34:30.788724 21702 raft_consensus.cc:348] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4216b1509b4913890b2a5c65b5c6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260620 04:34:30.789074 21702 raft_consensus.cc:374] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:30.789155 21702 raft_consensus.cc:729] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4216b1509b4913890b2a5c65b5c6b0, State: Initialized, Role: FOLLOWER
I20260620 04:34:30.789613 21702 consensus_queue.cc:260] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4216b1509b4913890b2a5c65b5c6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260620 04:34:30.789746 21702 raft_consensus.cc:388] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:30.789809 21702 raft_consensus.cc:482] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:30.789906 21702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 04:34:30 dist-test-slave-rzqm krb5kdc[21621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 04:34:30.791079 21702 raft_consensus.cc:504] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4216b1509b4913890b2a5c65b5c6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260620 04:34:30.791433 21702 leader_election.cc:302] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da4216b1509b4913890b2a5c65b5c6b0; no voters:
I20260620 04:34:30.791767 21702 leader_election.cc:288] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:30.791842 21707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:30.792053 21707 raft_consensus.cc:686] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [term 1 LEADER]: Becoming Leader. State: Replica: da4216b1509b4913890b2a5c65b5c6b0, State: Running, Role: LEADER
I20260620 04:34:30.792410 21707 consensus_queue.cc:237] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4216b1509b4913890b2a5c65b5c6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260620 04:34:30.793198 21702 sys_catalog.cc:564] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:30.794304 21709 sys_catalog.cc:455] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da4216b1509b4913890b2a5c65b5c6b0. Latest consensus state: current_term: 1 leader_uuid: "da4216b1509b4913890b2a5c65b5c6b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4216b1509b4913890b2a5c65b5c6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } } }
I20260620 04:34:30.794320 21708 sys_catalog.cc:455] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da4216b1509b4913890b2a5c65b5c6b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4216b1509b4913890b2a5c65b5c6b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } } }
I20260620 04:34:30.794515 21708 sys_catalog.cc:458] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:30.794785 21709 sys_catalog.cc:458] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:30.795094 21716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:30.797741 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:30.781994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:48715 (local address 127.0.0.1:41355)
0620 04:34:30.782504 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:30.782523 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:30.782560 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:34:30.783368 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:30.783393 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:30.783838 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:30.784199 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:30.784952 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:30.786505 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:30.787436 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:30.788198 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:30.788364 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:30.791404 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:30.791410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:30.791426 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:30.791458 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:30.793438 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:30.794177 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:30.794185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:30.794192 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:30.794539 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:30.795020 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:30.795023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:30.795025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:30.795577 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:30.795877 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:30.796496 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 04:34:30.796742 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":187,"threads_started":1}
I20260620 04:34:30.798144 21716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:30.801241 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:30.803858 21716 catalog_manager.cc:1347] Generated new cluster ID: a58b40d49968494ca9d72bbfd5b3e08f
I20260620 04:34:30.803925 21716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:30.812953 21716 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:30.813815 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:30.803349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:47521 (local address 127.0.0.1:41355)
0620 04:34:30.803463 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:30.803467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:30.805278 (+ 1811us) server_negotiation.cc:407] Connection header received
0620 04:34:30.805384 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:30.805387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:30.805441 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:30.805522 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:30.806262 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:30.807056 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:30.808121 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:30.808477 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:30.808571 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:30.810182 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:30.810186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:30.810189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:30.810214 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:30.811820 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:30.812332 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:30.812335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:30.812337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:30.812438 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:30.812806 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:30.812810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:30.812812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:30.813045 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:30.813170 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:30.813564 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 04:34:30.813662 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:34:30.813859 21716 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:30.825595 21716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da4216b1509b4913890b2a5c65b5c6b0: Generated new TSK 0
I20260620 04:34:30.826345 21716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:30.834501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 21634
W20260620 04:34:30.838973 21619 connection.cc:535] client connection to 127.0.0.1:41355 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41355): Connection reset by peer (error 104)
2026-06-20T04:34:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:30 dist-test-slave-rzqm krb5kdc[21733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:30 dist-test-slave-rzqm krb5kdc[21733](info): set up 2 sockets
Jun 20 04:34:30 dist-test-slave-rzqm krb5kdc[21733](info): commencing operation
krb5kdc: starting...
W20260620 04:34:31.788884 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:31 dist-test-slave-rzqm krb5kdc[21733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:31.901523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:32.009549 21746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:32.009912 21746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:32.009981 21746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:32.014185 21746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:32.014251 21746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:32.014271 21746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:32.014285 21746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:32.014299 21746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:32.019475 21746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:32.020457 21746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:32.021466 21746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:32.029488 21751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:32.029501 21752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:32.029623 21746 server_base.cc:1034] running on GCE node
W20260620 04:34:32.029495 21754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:32.030233 21746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:32.031355 21746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:32.032585 21746 hybrid_clock.cc:648] HybridClock initialized: now 1781930072032564 us; error 34 us; skew 500 ppm
Jun 20 04:34:32 dist-test-slave-rzqm krb5kdc[21733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:32.035449 21746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:32.036401 21746 webserver.cc:460] Webserver started at http://127.0.0.1:41933/ using document root <none> and password file <none>
I20260620 04:34:32.036916 21746 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:32.036972 21746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:32.037199 21746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:32.039254 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "af2511cdc75c44ddbf1d05dd685f0bcb"
format_stamp: "Formatted at 2026-06-20 04:34:32 on dist-test-slave-rzqm"
server_key: "d6a8e8bea264285a9d2dac0086d36bbd"
server_key_iv: "78f29c2c04d430f8cb3c3aa9a20a3ab3"
server_key_version: "encryptionkey@0"
I20260620 04:34:32.039779 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "af2511cdc75c44ddbf1d05dd685f0bcb"
format_stamp: "Formatted at 2026-06-20 04:34:32 on dist-test-slave-rzqm"
server_key: "d6a8e8bea264285a9d2dac0086d36bbd"
server_key_iv: "78f29c2c04d430f8cb3c3aa9a20a3ab3"
server_key_version: "encryptionkey@0"
I20260620 04:34:32.043606 21746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 04:34:32.046101 21761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:32.047499 21746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:32.047606 21746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "af2511cdc75c44ddbf1d05dd685f0bcb"
format_stamp: "Formatted at 2026-06-20 04:34:32 on dist-test-slave-rzqm"
server_key: "d6a8e8bea264285a9d2dac0086d36bbd"
server_key_iv: "78f29c2c04d430f8cb3c3aa9a20a3ab3"
server_key_version: "encryptionkey@0"
I20260620 04:34:32.047699 21746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:32.063611 21746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:32.066880 21746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:32.067112 21746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:32.074347 21746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46117
I20260620 04:34:32.074393 21813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46117 every 8 connection(s)
I20260620 04:34:32.075333 21746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:32.077484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 21746
I20260620 04:34:32.077620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:32.077860 392 external_mini_cluster.cc:1428] Setting key fc82c294884e0270b707862aacf94197
I20260620 04:34:32.078169 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:32.085338 21814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb: Bootstrap starting.
I20260620 04:34:32.088016 21814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:32.088114 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:32.079294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:39811 (local address 127.0.0.1:46117)
0620 04:34:32.079749 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:32.079769 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:32.079810 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:34:32.080537 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:32.080561 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:32.081049 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:32.081437 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:32.081853 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:32.083295 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:32.084276 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:32.084946 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:32.085152 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:32.086104 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:32.086111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:32.086127 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:32.086629 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:32.086700 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:32.086731 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:32.086861 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:32.087040 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 04:34:32.087180 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":133,"threads_started":1}
I20260620 04:34:32.088795 21814 log.cc:824] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:32.090873 21814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb: No bootstrap required, opened a new log
I20260620 04:34:32.091131 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:32.094506 21814 raft_consensus.cc:348] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2511cdc75c44ddbf1d05dd685f0bcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46117 } }
I20260620 04:34:32.094779 21814 raft_consensus.cc:374] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:32.094841 21814 raft_consensus.cc:729] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2511cdc75c44ddbf1d05dd685f0bcb, State: Initialized, Role: FOLLOWER
I20260620 04:34:32.095278 21814 consensus_queue.cc:260] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2511cdc75c44ddbf1d05dd685f0bcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46117 } }
I20260620 04:34:32.095410 21814 raft_consensus.cc:388] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:32.095501 21814 raft_consensus.cc:482] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:32.095595 21814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:32.096491 21814 raft_consensus.cc:504] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2511cdc75c44ddbf1d05dd685f0bcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46117 } }
I20260620 04:34:32.096820 21814 leader_election.cc:302] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af2511cdc75c44ddbf1d05dd685f0bcb; no voters:
I20260620 04:34:32.097148 21814 leader_election.cc:288] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:32.097225 21824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:32.097406 21824 raft_consensus.cc:686] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [term 1 LEADER]: Becoming Leader. State: Replica: af2511cdc75c44ddbf1d05dd685f0bcb, State: Running, Role: LEADER
I20260620 04:34:32.097769 21824 consensus_queue.cc:237] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2511cdc75c44ddbf1d05dd685f0bcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46117 } }
I20260620 04:34:32.098567 21814 sys_catalog.cc:564] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:32.098930 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:32.093058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:43825 (local address 127.0.0.1:46117)
0620 04:34:32.093174 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:32.093178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:32.095161 (+ 1983us) server_negotiation.cc:407] Connection header received
0620 04:34:32.095290 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:32.095292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:32.095333 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:32.095392 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:32.095660 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:32.096158 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:32.097027 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:32.097266 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:32.097375 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:32.098497 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:32.098501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:32.098504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:32.098518 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:32.098546 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:32.098556 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:32.098626 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:32.098756 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 04:34:32.098795 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:34:32.099716 21825 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader af2511cdc75c44ddbf1d05dd685f0bcb. Latest consensus state: current_term: 1 leader_uuid: "af2511cdc75c44ddbf1d05dd685f0bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2511cdc75c44ddbf1d05dd685f0bcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46117 } } }
I20260620 04:34:32.099994 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af2511cdc75c44ddbf1d05dd685f0bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2511cdc75c44ddbf1d05dd685f0bcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46117 } } }
I20260620 04:34:32.100117 21825 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:32.100142 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb [sys.catalog]: This master's current role is: LEADER
W20260620 04:34:32.103473 21839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:32.103623 21839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:32.103729 21835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:32.104411 21835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:32.110356 21835 catalog_manager.cc:1347] Generated new cluster ID: d7e92c5ee5974765beba1426f20d521d
I20260620 04:34:32.110441 21835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:32.116364 21835 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:32.117347 21835 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:32.124841 21835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af2511cdc75c44ddbf1d05dd685f0bcb: Generated new TSK 0
I20260620 04:34:32.125792 21835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:32.135160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 21746
W20260620 04:34:32.139727 21731 connection.cc:535] client connection to 127.0.0.1:46117 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46117): Connection reset by peer (error 104)
2026-06-20T04:34:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:32 dist-test-slave-rzqm krb5kdc[21845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:32 dist-test-slave-rzqm krb5kdc[21845](info): set up 2 sockets
Jun 20 04:34:32 dist-test-slave-rzqm krb5kdc[21845](info): commencing operation
krb5kdc: starting...
W20260620 04:34:33.085129 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:33 dist-test-slave-rzqm krb5kdc[21845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 04:34:33.204121 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:33.313091 21858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:33.313369 21858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:33.313411 21858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:33.317430 21858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:33.317493 21858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:33.317512 21858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:33.317530 21858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:33.317544 21858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:33.322907 21858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34463
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:33.323988 21858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:33.325133 21858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:33.333514 21864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:33.333804 21866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:33.333825 21863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:33.333570 21858 server_base.cc:1034] running on GCE node
I20260620 04:34:33.334510 21858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:33.335644 21858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:33.336820 21858 hybrid_clock.cc:648] HybridClock initialized: now 1781930073336802 us; error 30 us; skew 500 ppm
Jun 20 04:34:33 dist-test-slave-rzqm krb5kdc[21845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:33.339692 21858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 04:34:33.340622 21858 webserver.cc:460] Webserver started at http://127.0.0.1:37577/ using document root <none> and password file <none>
I20260620 04:34:33.341214 21858 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:33.341285 21858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:33.341485 21858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:33.343545 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "ca7d3715d5704a17ac4f50e885f1c4db"
format_stamp: "Formatted at 2026-06-20 04:34:33 on dist-test-slave-rzqm"
server_key: "817baaab0e78986c20fabc937fd44d85"
server_key_iv: "80f35585c3b54b69c00927d8a0684312"
server_key_version: "encryptionkey@0"
I20260620 04:34:33.344096 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca7d3715d5704a17ac4f50e885f1c4db"
format_stamp: "Formatted at 2026-06-20 04:34:33 on dist-test-slave-rzqm"
server_key: "817baaab0e78986c20fabc937fd44d85"
server_key_iv: "80f35585c3b54b69c00927d8a0684312"
server_key_version: "encryptionkey@0"
I20260620 04:34:33.347752 21858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 04:34:33.350126 21873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:33.351336 21858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:33.351451 21858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "ca7d3715d5704a17ac4f50e885f1c4db"
format_stamp: "Formatted at 2026-06-20 04:34:33 on dist-test-slave-rzqm"
server_key: "817baaab0e78986c20fabc937fd44d85"
server_key_iv: "80f35585c3b54b69c00927d8a0684312"
server_key_version: "encryptionkey@0"
I20260620 04:34:33.351550 21858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:33.356182 21858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:33.358999 21858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:33.359166 21858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:33.366096 21858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34463
I20260620 04:34:33.366099 21925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34463 every 8 connection(s)
I20260620 04:34:33.367100 21858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:33.369896 21926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:33.370023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 21858
I20260620 04:34:33.370158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:33.370466 392 external_mini_cluster.cc:1428] Setting key ab5180812452b2460ad096b955fe67af
I20260620 04:34:33.375792 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:33.371863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:59619 (local address 127.0.0.1:34463)
0620 04:34:33.372264 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:33.372278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:33.372315 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:34:33.373063 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:33.373083 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:33.373371 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:33.373677 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:33.374525 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:33.374530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:33.374543 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:33.374554 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:33.374609 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:33.374628 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:33.374735 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:33.374835 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 04:34:33.374981 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260620 04:34:33.376610 21926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db: Bootstrap starting.
I20260620 04:34:33.378618 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:33.379176 21926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:33.380043 21926 log.cc:824] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:33.382318 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:33.380823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:51223 (local address 127.0.0.1:34463)
0620 04:34:33.380936 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:33.380941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:33.381114 (+ 173us) server_negotiation.cc:407] Connection header received
0620 04:34:33.381205 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:33.381208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:33.381240 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:33.381311 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:33.381970 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:33.381974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:33.381976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:33.381988 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:33.382015 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:33.382028 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:33.382118 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:33.382168 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 04:34:33.382205 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:34:33.382370 21926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db: No bootstrap required, opened a new log
I20260620 04:34:33.386018 21926 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d3715d5704a17ac4f50e885f1c4db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34463 } }
I20260620 04:34:33.386315 21926 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:33.386376 21926 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7d3715d5704a17ac4f50e885f1c4db, State: Initialized, Role: FOLLOWER
I20260620 04:34:33.386812 21926 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d3715d5704a17ac4f50e885f1c4db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34463 } }
I20260620 04:34:33.386945 21926 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:33.387007 21926 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:33.387101 21926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:33.388010 21926 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d3715d5704a17ac4f50e885f1c4db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34463 } }
I20260620 04:34:33.388345 21926 leader_election.cc:302] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7d3715d5704a17ac4f50e885f1c4db; no voters:
I20260620 04:34:33.388653 21926 leader_election.cc:288] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:33.388790 21936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:33.388979 21936 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [term 1 LEADER]: Becoming Leader. State: Replica: ca7d3715d5704a17ac4f50e885f1c4db, State: Running, Role: LEADER
I20260620 04:34:33.389338 21936 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d3715d5704a17ac4f50e885f1c4db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34463 } }
I20260620 04:34:33.389854 21926 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:33.390924 21938 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7d3715d5704a17ac4f50e885f1c4db. Latest consensus state: current_term: 1 leader_uuid: "ca7d3715d5704a17ac4f50e885f1c4db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d3715d5704a17ac4f50e885f1c4db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34463 } } }
I20260620 04:34:33.391064 21938 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:33.391436 21945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:33.391398 21937 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca7d3715d5704a17ac4f50e885f1c4db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d3715d5704a17ac4f50e885f1c4db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34463 } } }
I20260620 04:34:33.391778 21937 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:33.394338 21945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:33.399976 21945 catalog_manager.cc:1347] Generated new cluster ID: 10f8449b442f42a6b4daacaa2d590b64
I20260620 04:34:33.400111 21945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:33.410872 21945 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:33.411250 21945 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:33.416903 21945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca7d3715d5704a17ac4f50e885f1c4db: Generated new TSK 0
I20260620 04:34:33.417723 21945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:33.425843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 21858
2026-06-20T04:34:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:33 dist-test-slave-rzqm krb5kdc[21957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:33 dist-test-slave-rzqm krb5kdc[21957](info): set up 2 sockets
Jun 20 04:34:33 dist-test-slave-rzqm krb5kdc[21957](info): commencing operation
krb5kdc: starting...
W20260620 04:34:34.377167 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:34 dist-test-slave-rzqm krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:34.490056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35593
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:34.600327 21970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:34.600610 21970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:34.600652 21970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:34.604661 21970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:34.604732 21970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:34.604750 21970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:34.604764 21970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:34.604779 21970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:34.610492 21970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:34.611604 21970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:34.612641 21970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:34.621347 21975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:34.621340 21978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:34.621388 21970 server_base.cc:1034] running on GCE node
W20260620 04:34:34.621349 21976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:34.622197 21970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:34.623396 21970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:34.624571 21970 hybrid_clock.cc:648] HybridClock initialized: now 1781930074624547 us; error 39 us; skew 500 ppm
Jun 20 04:34:34 dist-test-slave-rzqm krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:34.627393 21970 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:34.628276 21970 webserver.cc:460] Webserver started at http://127.0.98.62:42785/ using document root <none> and password file <none>
I20260620 04:34:34.628810 21970 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:34.628870 21970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:34.629087 21970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:34.631148 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "4399a4ee1ce34abba10da794b8749455"
format_stamp: "Formatted at 2026-06-20 04:34:34 on dist-test-slave-rzqm"
server_key: "ca3ea048538df3a67d66dc75d8a2d050"
server_key_iv: "11f2bd77cb09cdafb6c658a94c61d0de"
server_key_version: "encryptionkey@0"
I20260620 04:34:34.631666 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "4399a4ee1ce34abba10da794b8749455"
format_stamp: "Formatted at 2026-06-20 04:34:34 on dist-test-slave-rzqm"
server_key: "ca3ea048538df3a67d66dc75d8a2d050"
server_key_iv: "11f2bd77cb09cdafb6c658a94c61d0de"
server_key_version: "encryptionkey@0"
I20260620 04:34:34.635310 21970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 04:34:34.637859 21985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:34.639158 21970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 04:34:34.639276 21970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "4399a4ee1ce34abba10da794b8749455"
format_stamp: "Formatted at 2026-06-20 04:34:34 on dist-test-slave-rzqm"
server_key: "ca3ea048538df3a67d66dc75d8a2d050"
server_key_iv: "11f2bd77cb09cdafb6c658a94c61d0de"
server_key_version: "encryptionkey@0"
I20260620 04:34:34.639390 21970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:34.652889 21970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:34.655988 21970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:34.656172 21970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:34.663277 21970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35593
I20260620 04:34:34.663316 22037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35593 every 8 connection(s)
I20260620 04:34:34.664324 21970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:34.665895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 21970
I20260620 04:34:34.666034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:34.666325 392 external_mini_cluster.cc:1428] Setting key e0148a6279a7d98c574cf65ff288fa7a
I20260620 04:34:34.667687 22038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:34.674387 22038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455: Bootstrap starting.
Jun 20 04:34:34 dist-test-slave-rzqm krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:34.676810 22038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:34.677721 22038 log.cc:824] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:34.679944 22038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455: No bootstrap required, opened a new log
I20260620 04:34:34.681320 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:34.667709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47209 (local address 127.0.98.62:35593)
0620 04:34:34.668117 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:34.668141 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:34.668179 (+ 38us) server_negotiation.cc:407] Connection header received
0620 04:34:34.668826 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:34.668843 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:34.669247 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:34.669535 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:34.670420 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:34.671476 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:34.672626 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:34.673203 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:34.673324 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:34.676137 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:34.676143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:34.676157 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:34.676188 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:34.678081 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:34.678631 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:34.678636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:34.678641 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:34.678754 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:34.679180 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:34.679189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:34.679190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:34.679607 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:34.679765 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:34.680238 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 04:34:34.680482 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260620 04:34:34.683627 22038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a4ee1ce34abba10da794b8749455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260620 04:34:34.683852 22038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:34.683897 22038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4399a4ee1ce34abba10da794b8749455, State: Initialized, Role: FOLLOWER
I20260620 04:34:34.684262 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:34.684347 22038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a4ee1ce34abba10da794b8749455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260620 04:34:34.684497 22038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:34.684553 22038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:34.684652 22038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:34.685879 22038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a4ee1ce34abba10da794b8749455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260620 04:34:34.686244 22038 leader_election.cc:302] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4399a4ee1ce34abba10da794b8749455; no voters:
I20260620 04:34:34.686558 22038 leader_election.cc:288] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:34.686774 22047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:34.687009 22047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [term 1 LEADER]: Becoming Leader. State: Replica: 4399a4ee1ce34abba10da794b8749455, State: Running, Role: LEADER
I20260620 04:34:34.687366 22047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a4ee1ce34abba10da794b8749455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260620 04:34:34.687879 22038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:34.688478 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4399a4ee1ce34abba10da794b8749455. Latest consensus state: current_term: 1 leader_uuid: "4399a4ee1ce34abba10da794b8749455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a4ee1ce34abba10da794b8749455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } }
I20260620 04:34:34.689169 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:34.689062 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4399a4ee1ce34abba10da794b8749455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a4ee1ce34abba10da794b8749455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } }
I20260620 04:34:34.689488 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:34.689849 22060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:34.692886 22060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:34.696245 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:34.686542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40209 (local address 127.0.98.62:35593)
0620 04:34:34.686650 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:34.686655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:34.686897 (+ 242us) server_negotiation.cc:407] Connection header received
0620 04:34:34.687090 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:34.687093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:34.687149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:34.687233 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:34.688122 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:34.688869 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:34.690011 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:34.690346 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:34.690446 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:34.692638 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:34.692643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:34.692645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:34.692671 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:34.694269 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:34.694843 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:34.694847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:34.694849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:34.694936 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:34.695329 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:34.695333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:34.695335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:34.695563 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:34.695704 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:34.695998 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 04:34:34.696101 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 04:34:34.702133 22060 catalog_manager.cc:1347] Generated new cluster ID: 89e102cda834485f9451257a4bf461b3
I20260620 04:34:34.702255 22060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:34.710598 22060 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:34.711781 22060 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:34.727617 22060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4399a4ee1ce34abba10da794b8749455: Generated new TSK 0
I20260620 04:34:34.728497 22060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:34.733403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 21970
W20260620 04:34:34.738062 21955 connection.cc:535] client connection to 127.0.98.62:35593 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35593): Connection reset by peer (error 104)
2026-06-20T04:34:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:34 dist-test-slave-rzqm krb5kdc[22069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:34 dist-test-slave-rzqm krb5kdc[22069](info): set up 2 sockets
Jun 20 04:34:34 dist-test-slave-rzqm krb5kdc[22069](info): commencing operation
krb5kdc: starting...
W20260620 04:34:35.680991 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:35 dist-test-slave-rzqm krb5kdc[22069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:35.797977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:35.905587 22082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:35.905869 22082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:35.905910 22082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:35.909945 22082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:35.910010 22082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:35.910029 22082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:35.910046 22082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:35.910060 22082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:35.915429 22082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:35.916458 22082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:35.917474 22082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:35.925681 22090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:35.925673 22087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:35.925726 22082 server_base.cc:1034] running on GCE node
W20260620 04:34:35.925681 22088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:35.926645 22082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:35.927853 22082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:35.929044 22082 hybrid_clock.cc:648] HybridClock initialized: now 1781930075929008 us; error 51 us; skew 500 ppm
Jun 20 04:34:35 dist-test-slave-rzqm krb5kdc[22069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:35.936297 22082 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:35.937335 22082 webserver.cc:460] Webserver started at http://127.0.98.62:43503/ using document root <none> and password file <none>
I20260620 04:34:35.937891 22082 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:35.937952 22082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:35.938148 22082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:35.940209 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "2a11cc8df83948a583491e4e2f9e35fb"
format_stamp: "Formatted at 2026-06-20 04:34:35 on dist-test-slave-rzqm"
server_key: "851733d9246bb81b4cb3ef23ecf36d46"
server_key_iv: "5d417749394a17d6d05d5f724ea8515e"
server_key_version: "encryptionkey@0"
I20260620 04:34:35.940743 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a11cc8df83948a583491e4e2f9e35fb"
format_stamp: "Formatted at 2026-06-20 04:34:35 on dist-test-slave-rzqm"
server_key: "851733d9246bb81b4cb3ef23ecf36d46"
server_key_iv: "5d417749394a17d6d05d5f724ea8515e"
server_key_version: "encryptionkey@0"
I20260620 04:34:35.944648 22082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:34:35.947244 22097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:35.948486 22082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:34:35.948606 22082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "2a11cc8df83948a583491e4e2f9e35fb"
format_stamp: "Formatted at 2026-06-20 04:34:35 on dist-test-slave-rzqm"
server_key: "851733d9246bb81b4cb3ef23ecf36d46"
server_key_iv: "5d417749394a17d6d05d5f724ea8515e"
server_key_version: "encryptionkey@0"
I20260620 04:34:35.948719 22082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:35.961220 22082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:35.964612 22082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:35.964802 22082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:35.972005 22082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43377
I20260620 04:34:35.972024 22149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43377 every 8 connection(s)
I20260620 04:34:35.973021 22082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:35.973958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 22082
I20260620 04:34:35.974076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:35.974306 392 external_mini_cluster.cc:1428] Setting key af3d19f30e4192316699c509c6d9476c
I20260620 04:34:35.976056 22150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:35.983206 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb: Bootstrap starting.
I20260620 04:34:35.984808 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:35.975774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56207 (local address 127.0.98.62:43377)
0620 04:34:35.976203 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:35.976223 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:35.976264 (+ 41us) server_negotiation.cc:407] Connection header received
0620 04:34:35.977124 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:35.977149 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:35.977595 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:35.977993 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:35.978466 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:35.979988 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:35.980974 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:35.981714 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:35.981879 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:35.982832 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:35.982840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:35.982858 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:35.983354 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:35.983424 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:35.983453 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:35.983579 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:35.983726 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 04:34:35.983864 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":106,"threads_started":1}
I20260620 04:34:35.985839 22150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:35.986698 22150 log.cc:824] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:35.988047 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:35.988903 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb: No bootstrap required, opened a new log
I20260620 04:34:35.992946 22150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a11cc8df83948a583491e4e2f9e35fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260620 04:34:35.993316 22150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:35.993418 22150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a11cc8df83948a583491e4e2f9e35fb, State: Initialized, Role: FOLLOWER
I20260620 04:34:35.993949 22150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a11cc8df83948a583491e4e2f9e35fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260620 04:34:35.994136 22150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:35.994191 22150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:35.994326 22150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:35.994783 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:35.990369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44975 (local address 127.0.98.62:43377)
0620 04:34:35.990498 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:35.990502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:35.990628 (+ 126us) server_negotiation.cc:407] Connection header received
0620 04:34:35.990723 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:35.990726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:35.990771 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:35.990857 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:35.991306 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:35.992067 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:35.992908 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:35.993248 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:35.993358 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:35.994300 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:35.994304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:35.994306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:35.994321 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:35.994352 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:35.994365 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:35.994455 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:35.994607 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 04:34:35.994647 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:34:35.995527 22150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a11cc8df83948a583491e4e2f9e35fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260620 04:34:35.995908 22150 leader_election.cc:302] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a11cc8df83948a583491e4e2f9e35fb; no voters:
I20260620 04:34:35.996224 22150 leader_election.cc:288] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:35.996336 22160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:35.996549 22160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [term 1 LEADER]: Becoming Leader. State: Replica: 2a11cc8df83948a583491e4e2f9e35fb, State: Running, Role: LEADER
I20260620 04:34:35.996883 22160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a11cc8df83948a583491e4e2f9e35fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260620 04:34:35.997460 22150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:35.998528 22162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a11cc8df83948a583491e4e2f9e35fb. Latest consensus state: current_term: 1 leader_uuid: "2a11cc8df83948a583491e4e2f9e35fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a11cc8df83948a583491e4e2f9e35fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } }
I20260620 04:34:35.998667 22162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:35.999186 22169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:35.999305 22161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a11cc8df83948a583491e4e2f9e35fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a11cc8df83948a583491e4e2f9e35fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } }
I20260620 04:34:35.999461 22161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:36.002399 22169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:36.007736 22169 catalog_manager.cc:1347] Generated new cluster ID: 60389626e7bc4e4dbe7f3160f58bc3c8
I20260620 04:34:36.007826 22169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:36.022630 22169 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:36.023999 22169 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:36.035218 22169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a11cc8df83948a583491e4e2f9e35fb: Generated new TSK 0
I20260620 04:34:36.036144 22169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:36.045135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 22082
W20260620 04:34:36.049970 22067 connection.cc:535] client connection to 127.0.98.62:43377 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43377): Connection reset by peer (error 104)
2026-06-20T04:34:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:36 dist-test-slave-rzqm krb5kdc[22181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:36 dist-test-slave-rzqm krb5kdc[22181](info): set up 2 sockets
Jun 20 04:34:36 dist-test-slave-rzqm krb5kdc[22181](info): commencing operation
krb5kdc: starting...
W20260620 04:34:37.001055 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:37 dist-test-slave-rzqm krb5kdc[22181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:37.124495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39085
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:37.234874 22194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:37.235185 22194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:37.235255 22194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:37.239427 22194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:37.239509 22194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:37.239537 22194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:37.239559 22194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:37.239588 22194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:37.245194 22194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39085
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:37.246323 22194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:37.247355 22194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:37.255921 22200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:37.256007 22194 server_base.cc:1034] running on GCE node
W20260620 04:34:37.255904 22199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:37.256183 22202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:37.256763 22194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:37.257983 22194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:37.259214 22194 hybrid_clock.cc:648] HybridClock initialized: now 1781930077259142 us; error 38 us; skew 500 ppm
Jun 20 04:34:37 dist-test-slave-rzqm krb5kdc[22181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:37.262224 22194 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:37.263209 22194 webserver.cc:460] Webserver started at http://127.0.98.62:39399/ using document root <none> and password file <none>
I20260620 04:34:37.263800 22194 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:37.263877 22194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:37.264098 22194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:37.266307 22194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "01f3e46b86d7475fb4990b96b99d57f7"
format_stamp: "Formatted at 2026-06-20 04:34:37 on dist-test-slave-rzqm"
server_key: "78e4d770e86889776c5308c6e704e76b"
server_key_iv: "a96bef8ffbe97b18d95812739aa75b5a"
server_key_version: "encryptionkey@0"
I20260620 04:34:37.266901 22194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "01f3e46b86d7475fb4990b96b99d57f7"
format_stamp: "Formatted at 2026-06-20 04:34:37 on dist-test-slave-rzqm"
server_key: "78e4d770e86889776c5308c6e704e76b"
server_key_iv: "a96bef8ffbe97b18d95812739aa75b5a"
server_key_version: "encryptionkey@0"
I20260620 04:34:37.271013 22194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 04:34:37.273695 22209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:37.275020 22194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:37.275166 22194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "01f3e46b86d7475fb4990b96b99d57f7"
format_stamp: "Formatted at 2026-06-20 04:34:37 on dist-test-slave-rzqm"
server_key: "78e4d770e86889776c5308c6e704e76b"
server_key_iv: "a96bef8ffbe97b18d95812739aa75b5a"
server_key_version: "encryptionkey@0"
I20260620 04:34:37.275274 22194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:37.279892 22194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:37.282867 22194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:37.283044 22194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:37.290123 22194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39085
I20260620 04:34:37.290189 22261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39085 every 8 connection(s)
I20260620 04:34:37.291123 22194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:37.293906 22262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:37.299382 22262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7: Bootstrap starting.
I20260620 04:34:37.300843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 22194
I20260620 04:34:37.300976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:37.301241 392 external_mini_cluster.cc:1428] Setting key 52cefd5ac242a35d467922eccd2ecd41
I20260620 04:34:37.302021 22262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:37.302847 22262 log.cc:824] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:37.304991 22262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7: No bootstrap required, opened a new log
I20260620 04:34:37.306443 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:37.302315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50851 (local address 127.0.98.62:39085)
0620 04:34:37.302797 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:37.302814 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:37.302857 (+ 43us) server_negotiation.cc:407] Connection header received
0620 04:34:37.303644 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:37.303670 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:37.304018 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:37.304373 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:37.305047 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:37.305053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:37.305069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:37.305081 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:37.305143 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:37.305171 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:37.305277 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:37.305372 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 04:34:37.305536 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":135,"threads_started":1}
I20260620 04:34:37.309041 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:37.308868 22262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f3e46b86d7475fb4990b96b99d57f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39085 } }
I20260620 04:34:37.309146 22262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:37.309199 22262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f3e46b86d7475fb4990b96b99d57f7, State: Initialized, Role: FOLLOWER
I20260620 04:34:37.309669 22262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f3e46b86d7475fb4990b96b99d57f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39085 } }
I20260620 04:34:37.309834 22262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:37.309912 22262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:37.310009 22262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:37.311182 22262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f3e46b86d7475fb4990b96b99d57f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39085 } }
I20260620 04:34:37.311527 22262 leader_election.cc:302] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01f3e46b86d7475fb4990b96b99d57f7; no voters:
I20260620 04:34:37.311884 22262 leader_election.cc:288] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:37.311935 22272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:37.312129 22272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [term 1 LEADER]: Becoming Leader. State: Replica: 01f3e46b86d7475fb4990b96b99d57f7, State: Running, Role: LEADER
I20260620 04:34:37.312410 22272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f3e46b86d7475fb4990b96b99d57f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39085 } }
I20260620 04:34:37.312933 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:37.311245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39231 (local address 127.0.98.62:39085)
0620 04:34:37.311357 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:37.311361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:37.311546 (+ 185us) server_negotiation.cc:407] Connection header received
0620 04:34:37.311644 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:37.311647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:37.311684 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:37.311764 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:37.312429 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:37.312433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:37.312436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:37.312447 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:37.312479 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:37.312492 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:37.312564 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:37.312807 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 04:34:37.312842 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:34:37.313205 22262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:37.314213 22274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01f3e46b86d7475fb4990b96b99d57f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f3e46b86d7475fb4990b96b99d57f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39085 } } }
I20260620 04:34:37.314338 22274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:37.314708 22281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:37.314677 22273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01f3e46b86d7475fb4990b96b99d57f7. Latest consensus state: current_term: 1 leader_uuid: "01f3e46b86d7475fb4990b96b99d57f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01f3e46b86d7475fb4990b96b99d57f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39085 } } }
I20260620 04:34:37.317060 22273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:37.318055 22281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:37.323889 22281 catalog_manager.cc:1347] Generated new cluster ID: 891cf42138484f8494e6bd6e6ff44011
I20260620 04:34:37.323968 22281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:37.334445 22281 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:37.334798 22281 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:37.355427 22281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01f3e46b86d7475fb4990b96b99d57f7: Generated new TSK 0
I20260620 04:34:37.356206 22281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:37.365628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 22194
2026-06-20T04:34:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:37 dist-test-slave-rzqm krb5kdc[22293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:37 dist-test-slave-rzqm krb5kdc[22293](info): set up 2 sockets
Jun 20 04:34:37 dist-test-slave-rzqm krb5kdc[22293](info): commencing operation
krb5kdc: starting...
W20260620 04:34:38.329146 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:38 dist-test-slave-rzqm krb5kdc[22293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:38.445886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:38.558571 22306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:38.558874 22306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:38.558936 22306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:38.563040 22306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:38.563117 22306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:38.563151 22306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:38.563174 22306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:38.563194 22306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:38.569053 22306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:38.570240 22306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:38.571301 22306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:38.580082 22314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:38.580116 22311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:38.580288 22306 server_base.cc:1034] running on GCE node
W20260620 04:34:38.580188 22312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:38.580921 22306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:38.582090 22306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:38.583273 22306 hybrid_clock.cc:648] HybridClock initialized: now 1781930078583243 us; error 41 us; skew 500 ppm
Jun 20 04:34:38 dist-test-slave-rzqm krb5kdc[22293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:38.585922 22306 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:38.586838 22306 webserver.cc:460] Webserver started at http://127.0.98.62:42003/ using document root <none> and password file <none>
I20260620 04:34:38.587374 22306 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:38.587440 22306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:38.587623 22306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:38.589681 22306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "15366a1842064121bd33e2ce41be8f13"
format_stamp: "Formatted at 2026-06-20 04:34:38 on dist-test-slave-rzqm"
server_key: "2caffdb77d41a7b5e58e8b5d2a7a2dfe"
server_key_iv: "a2626e9c3bbbe1abff1ea8c64dcf864a"
server_key_version: "encryptionkey@0"
I20260620 04:34:38.590220 22306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "15366a1842064121bd33e2ce41be8f13"
format_stamp: "Formatted at 2026-06-20 04:34:38 on dist-test-slave-rzqm"
server_key: "2caffdb77d41a7b5e58e8b5d2a7a2dfe"
server_key_iv: "a2626e9c3bbbe1abff1ea8c64dcf864a"
server_key_version: "encryptionkey@0"
I20260620 04:34:38.593921 22306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 04:34:38.596411 22321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:38.597704 22306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:38.597824 22306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "15366a1842064121bd33e2ce41be8f13"
format_stamp: "Formatted at 2026-06-20 04:34:38 on dist-test-slave-rzqm"
server_key: "2caffdb77d41a7b5e58e8b5d2a7a2dfe"
server_key_iv: "a2626e9c3bbbe1abff1ea8c64dcf864a"
server_key_version: "encryptionkey@0"
I20260620 04:34:38.597930 22306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:38.612452 22306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:38.616012 22306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:38.616230 22306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:38.623000 22306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36115
I20260620 04:34:38.623034 22373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36115 every 8 connection(s)
I20260620 04:34:38.623955 22306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:38.626753 22374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:38.632036 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 22306
I20260620 04:34:38.632192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:38.632464 392 external_mini_cluster.cc:1428] Setting key 0685d79d576b8d9fcfa4a177005007d4
I20260620 04:34:38.632511 22374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13: Bootstrap starting.
I20260620 04:34:38.635080 22374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:38.635805 22374 log.cc:824] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:38.637756 22374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13: No bootstrap required, opened a new log
Jun 20 04:34:38 dist-test-slave-rzqm krb5kdc[22293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:38.641611 22374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15366a1842064121bd33e2ce41be8f13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } }
I20260620 04:34:38.641888 22374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:38.641957 22374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15366a1842064121bd33e2ce41be8f13, State: Initialized, Role: FOLLOWER
I20260620 04:34:38.642453 22374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15366a1842064121bd33e2ce41be8f13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } }
I20260620 04:34:38.642622 22374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:38.642701 22374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:38.642792 22374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:38.643926 22374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15366a1842064121bd33e2ce41be8f13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } }
I20260620 04:34:38.644272 22374 leader_election.cc:302] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15366a1842064121bd33e2ce41be8f13; no voters:
I20260620 04:34:38.644663 22374 leader_election.cc:288] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:38.644732 22379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:38.644920 22379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [term 1 LEADER]: Becoming Leader. State: Replica: 15366a1842064121bd33e2ce41be8f13, State: Running, Role: LEADER
I20260620 04:34:38.645313 22379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15366a1842064121bd33e2ce41be8f13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } }
I20260620 04:34:38.646039 22374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:38.647135 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:38.633862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36199 (local address 127.0.98.62:36115)
0620 04:34:38.634304 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:38.634319 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:38.634355 (+ 36us) server_negotiation.cc:407] Connection header received
0620 04:34:38.635018 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:38.635037 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:38.635420 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:38.635737 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:38.636566 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:38.637618 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:38.638490 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:38.638971 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:38.639126 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:38.641849 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:38.641856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:38.641874 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:38.641908 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:38.643816 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:38.644243 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:38.644249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:38.644255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:38.644437 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:38.644700 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:38.644704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:38.644706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:38.645327 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:38.645527 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:38.645821 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 04:34:38.646153 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":137,"threads_started":1}
I20260620 04:34:38.647094 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15366a1842064121bd33e2ce41be8f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15366a1842064121bd33e2ce41be8f13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } }
I20260620 04:34:38.647632 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:38.647305 22380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15366a1842064121bd33e2ce41be8f13. Latest consensus state: current_term: 1 leader_uuid: "15366a1842064121bd33e2ce41be8f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15366a1842064121bd33e2ce41be8f13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } }
I20260620 04:34:38.647974 22380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:38.650897 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 04:34:38.651000 22394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:38.651085 22394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:38.651165 22391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:38.652094 22391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:38.658336 22391 catalog_manager.cc:1347] Generated new cluster ID: 72fca29376654df287bd2634ef413755
I20260620 04:34:38.658420 22391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:38.662209 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:38.653433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48551 (local address 127.0.98.62:36115)
0620 04:34:38.653544 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:38.653549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:38.654348 (+ 799us) server_negotiation.cc:407] Connection header received
0620 04:34:38.654473 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:38.654477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:38.654529 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:38.654619 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:38.655423 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:38.656243 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:38.657293 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:38.657660 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:38.657768 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:38.659248 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:38.659252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:38.659255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:38.659283 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:38.660604 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:38.661048 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:38.661051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:38.661052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:38.661134 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:38.661385 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:38.661389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:38.661391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:38.661569 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:38.661685 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:38.661921 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 04:34:38.662036 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 04:34:38.671155 22391 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:38.672165 22391 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:38.682561 22391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15366a1842064121bd33e2ce41be8f13: Generated new TSK 0
I20260620 04:34:38.683488 22391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:38.688165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 22306
2026-06-20T04:34:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:38 dist-test-slave-rzqm krb5kdc[22405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:38 dist-test-slave-rzqm krb5kdc[22405](info): set up 2 sockets
Jun 20 04:34:38 dist-test-slave-rzqm krb5kdc[22405](info): commencing operation
krb5kdc: starting...
W20260620 04:34:39.637228 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:39 dist-test-slave-rzqm krb5kdc[22405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:39.752405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34269
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:39.861774 22418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:39.862083 22418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:39.862145 22418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:39.866221 22418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:39.866304 22418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:39.866339 22418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:39.866360 22418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:39.866389 22418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:39.871915 22418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:39.873063 22418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:39.874105 22418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:39.882921 22426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:39.882921 22424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:39.882995 22423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:39.883144 22418 server_base.cc:1034] running on GCE node
I20260620 04:34:39.883903 22418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:39.885166 22418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:39.886338 22418 hybrid_clock.cc:648] HybridClock initialized: now 1781930079886308 us; error 46 us; skew 500 ppm
Jun 20 04:34:39 dist-test-slave-rzqm krb5kdc[22405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:39.889212 22418 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:39.890239 22418 webserver.cc:460] Webserver started at http://127.0.98.62:42257/ using document root <none> and password file <none>
I20260620 04:34:39.890807 22418 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:39.890872 22418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:39.891064 22418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:39.893216 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "d9958d48f2764b5fb7eab11e05976d93"
format_stamp: "Formatted at 2026-06-20 04:34:39 on dist-test-slave-rzqm"
server_key: "6ecc90320e969258d5cbb07e80c7caeb"
server_key_iv: "44178d01ea91ad7ea6617fd83c793057"
server_key_version: "encryptionkey@0"
I20260620 04:34:39.893781 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9958d48f2764b5fb7eab11e05976d93"
format_stamp: "Formatted at 2026-06-20 04:34:39 on dist-test-slave-rzqm"
server_key: "6ecc90320e969258d5cbb07e80c7caeb"
server_key_iv: "44178d01ea91ad7ea6617fd83c793057"
server_key_version: "encryptionkey@0"
I20260620 04:34:39.897632 22418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 04:34:39.900264 22433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:39.901633 22418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:39.901755 22418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "d9958d48f2764b5fb7eab11e05976d93"
format_stamp: "Formatted at 2026-06-20 04:34:39 on dist-test-slave-rzqm"
server_key: "6ecc90320e969258d5cbb07e80c7caeb"
server_key_iv: "44178d01ea91ad7ea6617fd83c793057"
server_key_version: "encryptionkey@0"
I20260620 04:34:39.901858 22418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:39.930315 22418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:39.933670 22418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:39.933876 22418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:39.940734 22418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34269
I20260620 04:34:39.940773 22485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34269 every 8 connection(s)
I20260620 04:34:39.941732 22418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:39.944621 22486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:39.949048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 22418
I20260620 04:34:39.949172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:39.949419 392 external_mini_cluster.cc:1428] Setting key 44e6ba1824bcb872ffe19a54aaede0c1
I20260620 04:34:39.950382 22486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93: Bootstrap starting.
I20260620 04:34:39.952775 22486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:39.953575 22486 log.cc:824] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:39.955588 22486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93: No bootstrap required, opened a new log
I20260620 04:34:39.958488 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:39.950778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43407 (local address 127.0.98.62:34269)
0620 04:34:39.951258 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:39.951274 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:39.951311 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:34:39.951951 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:39.951968 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:39.952351 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:39.952673 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:39.953142 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:39.954161 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:39.955184 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:39.955667 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:39.955806 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:39.956681 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:39.956686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:39.956702 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:39.957160 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:39.957245 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:39.957271 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:39.957345 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:39.957487 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 04:34:39.957685 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":135,"threads_started":1}
I20260620 04:34:39.959600 22486 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9958d48f2764b5fb7eab11e05976d93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 04:34:39.959900 22486 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:39.959995 22486 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9958d48f2764b5fb7eab11e05976d93, State: Initialized, Role: FOLLOWER
I20260620 04:34:39.960470 22486 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9958d48f2764b5fb7eab11e05976d93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 04:34:39.960635 22486 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:39.960700 22486 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:39.960822 22486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:39.961846 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:39.961939 22486 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9958d48f2764b5fb7eab11e05976d93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 04:34:39.962292 22486 leader_election.cc:302] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9958d48f2764b5fb7eab11e05976d93; no voters:
I20260620 04:34:39.962682 22486 leader_election.cc:288] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:39.962783 22491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:39.962984 22491 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [term 1 LEADER]: Becoming Leader. State: Replica: d9958d48f2764b5fb7eab11e05976d93, State: Running, Role: LEADER
I20260620 04:34:39.963352 22491 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9958d48f2764b5fb7eab11e05976d93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } }
I20260620 04:34:39.964062 22486 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:39.964929 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9958d48f2764b5fb7eab11e05976d93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9958d48f2764b5fb7eab11e05976d93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } } }
I20260620 04:34:39.965101 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:39.965342 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9958d48f2764b5fb7eab11e05976d93. Latest consensus state: current_term: 1 leader_uuid: "d9958d48f2764b5fb7eab11e05976d93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9958d48f2764b5fb7eab11e05976d93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34269 } } }
I20260620 04:34:39.965420 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:39.966475 22505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:39.968751 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:39.964021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51131 (local address 127.0.98.62:34269)
0620 04:34:39.964128 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:39.964133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:39.964446 (+ 313us) server_negotiation.cc:407] Connection header received
0620 04:34:39.964637 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:39.964641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:39.964693 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:39.964744 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:39.965216 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:39.965952 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:39.967106 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:39.967337 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:39.967423 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:39.968286 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:39.968290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:39.968293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:39.968306 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:39.968336 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:39.968351 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:39.968395 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:39.968555 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 04:34:39.968614 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 04:34:39.969436 22505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:39.974741 22505 catalog_manager.cc:1347] Generated new cluster ID: d985f0d39c204a4da1a3a0c73e285453
I20260620 04:34:39.974802 22505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:39.984179 22505 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:39.985198 22505 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:39.992388 22505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9958d48f2764b5fb7eab11e05976d93: Generated new TSK 0
I20260620 04:34:39.993284 22505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:40.006619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 22418
2026-06-20T04:34:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:40 dist-test-slave-rzqm krb5kdc[22517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:40 dist-test-slave-rzqm krb5kdc[22517](info): set up 2 sockets
Jun 20 04:34:40 dist-test-slave-rzqm krb5kdc[22517](info): commencing operation
krb5kdc: starting...
W20260620 04:34:40.965070 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:41 dist-test-slave-rzqm krb5kdc[22517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:41.083402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46321
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:41.193908 22530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:41.194185 22530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:41.194227 22530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:41.198195 22530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:41.198256 22530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:41.198274 22530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:41.198289 22530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:41.198331 22530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:41.203653 22530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:41.204742 22530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:41.205799 22530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:34:41.214495 22530 server_base.cc:1034] running on GCE node
W20260620 04:34:41.214524 22535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:41.214687 22536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:41.214699 22538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:41.215281 22530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:41.216419 22530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:41.217579 22530 hybrid_clock.cc:648] HybridClock initialized: now 1781930081217563 us; error 37 us; skew 500 ppm
Jun 20 04:34:41 dist-test-slave-rzqm krb5kdc[22517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:41.225469 22530 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:41.226446 22530 webserver.cc:460] Webserver started at http://127.0.98.62:36927/ using document root <none> and password file <none>
I20260620 04:34:41.226989 22530 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:41.227049 22530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:41.227224 22530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:41.229363 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "70f50c6a6955406ab96dfdad802eb4cc"
format_stamp: "Formatted at 2026-06-20 04:34:41 on dist-test-slave-rzqm"
server_key: "68f65aadf319797cd40b4c79941f9696"
server_key_iv: "52b2719da7c8c5ccaad72de52d51b62e"
server_key_version: "encryptionkey@0"
I20260620 04:34:41.229893 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "70f50c6a6955406ab96dfdad802eb4cc"
format_stamp: "Formatted at 2026-06-20 04:34:41 on dist-test-slave-rzqm"
server_key: "68f65aadf319797cd40b4c79941f9696"
server_key_iv: "52b2719da7c8c5ccaad72de52d51b62e"
server_key_version: "encryptionkey@0"
I20260620 04:34:41.233600 22530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 04:34:41.236039 22545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:41.237322 22530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:41.237430 22530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "70f50c6a6955406ab96dfdad802eb4cc"
format_stamp: "Formatted at 2026-06-20 04:34:41 on dist-test-slave-rzqm"
server_key: "68f65aadf319797cd40b4c79941f9696"
server_key_iv: "52b2719da7c8c5ccaad72de52d51b62e"
server_key_version: "encryptionkey@0"
I20260620 04:34:41.237524 22530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:41.242143 22530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:41.245151 22530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:41.245306 22530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:41.252249 22530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46321
I20260620 04:34:41.252264 22597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46321 every 8 connection(s)
I20260620 04:34:41.253302 22530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:41.256178 22598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:41.259467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 22530
I20260620 04:34:41.259618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:41.259882 392 external_mini_cluster.cc:1428] Setting key 42dc7087d9335356fe216653be35bcbc
I20260620 04:34:41.262864 22598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc: Bootstrap starting.
I20260620 04:34:41.265363 22598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:41.265626 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:41.261433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47959 (local address 127.0.98.62:46321)
0620 04:34:41.261962 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:41.261983 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:41.262030 (+ 47us) server_negotiation.cc:407] Connection header received
0620 04:34:41.262937 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:41.262961 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:41.263265 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:41.263592 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:41.264373 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:41.264378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:41.264390 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:41.264400 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:41.264455 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:41.264474 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:41.264598 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:41.264669 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 04:34:41.264838 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":149,"threads_started":1}
I20260620 04:34:41.266264 22598 log.cc:824] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:41.268402 22598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc: No bootstrap required, opened a new log
I20260620 04:34:41.268631 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:41.272161 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:41.270606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45803 (local address 127.0.98.62:46321)
0620 04:34:41.270730 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:41.270734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:41.270917 (+ 183us) server_negotiation.cc:407] Connection header received
0620 04:34:41.271023 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:41.271026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:41.271057 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:41.271133 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:41.271820 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:41.271824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:41.271826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 04:34:41.271837 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:41.271860 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 04:34:41.271870 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:41.271938 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:41.272009 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 04:34:41.272046 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 04:34:41.272301 22598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f50c6a6955406ab96dfdad802eb4cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } }
I20260620 04:34:41.272586 22598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 04:34:41.272643 22598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f50c6a6955406ab96dfdad802eb4cc, State: Initialized, Role: FOLLOWER
I20260620 04:34:41.273135 22598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f50c6a6955406ab96dfdad802eb4cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } }
I20260620 04:34:41.273273 22598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 04:34:41.273336 22598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 04:34:41.273432 22598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:41.274343 22598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f50c6a6955406ab96dfdad802eb4cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } }
I20260620 04:34:41.274665 22598 leader_election.cc:302] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f50c6a6955406ab96dfdad802eb4cc; no voters:
I20260620 04:34:41.274973 22598 leader_election.cc:288] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 04:34:41.275094 22608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:41.275309 22608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [term 1 LEADER]: Becoming Leader. State: Replica: 70f50c6a6955406ab96dfdad802eb4cc, State: Running, Role: LEADER
I20260620 04:34:41.275648 22608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f50c6a6955406ab96dfdad802eb4cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } }
I20260620 04:34:41.276131 22598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:41.277297 22609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70f50c6a6955406ab96dfdad802eb4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f50c6a6955406ab96dfdad802eb4cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } } }
I20260620 04:34:41.277436 22609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:41.277753 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f50c6a6955406ab96dfdad802eb4cc. Latest consensus state: current_term: 1 leader_uuid: "70f50c6a6955406ab96dfdad802eb4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f50c6a6955406ab96dfdad802eb4cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } } }
I20260620 04:34:41.277875 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc [sys.catalog]: This master's current role is: LEADER
W20260620 04:34:41.280593 22623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:41.280664 22623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:41.280745 22617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:41.281483 22617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:41.287482 22617 catalog_manager.cc:1347] Generated new cluster ID: d73d08f639424ee284088c813277be5b
I20260620 04:34:41.287567 22617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:41.308904 22617 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:41.309329 22617 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:41.316526 22617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70f50c6a6955406ab96dfdad802eb4cc: Generated new TSK 0
I20260620 04:34:41.317392 22617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 04:34:41.328619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 22530
2026-06-20T04:34:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1322 ms)
[----------] 18 tests from AuthTokenIssuingTest (23596 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:41 dist-test-slave-rzqm krb5kdc[22629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:41 dist-test-slave-rzqm krb5kdc[22629](info): set up 2 sockets
Jun 20 04:34:41 dist-test-slave-rzqm krb5kdc[22629](info): commencing operation
krb5kdc: starting...
W20260620 04:34:42.281222 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:42 dist-test-slave-rzqm krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:42.395483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41115,127.0.98.61:45879,127.0.98.60:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:42.503509 22642 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:34:42.503793 22642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:42.503815 22642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:42.503863 22642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:42.508576 22642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:42.508659 22642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:42.508688 22642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:42.508710 22642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:42.508737 22642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:42.514456 22642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41115,127.0.98.61:45879,127.0.98.60:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:42.515901 22642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:42.517274 22642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:42.526187 22650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:42.526232 22642 server_base.cc:1034] running on GCE node
W20260620 04:34:42.526203 22648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:42.526193 22647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:42.527026 22642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:42.528175 22642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:42.529368 22642 hybrid_clock.cc:648] HybridClock initialized: now 1781930082529339 us; error 44 us; skew 500 ppm
Jun 20 04:34:42 dist-test-slave-rzqm krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:42.532301 22642 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:42.533239 22642 webserver.cc:460] Webserver started at http://127.0.98.62:37099/ using document root <none> and password file <none>
I20260620 04:34:42.533798 22642 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:42.533869 22642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:42.534057 22642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:42.536136 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "6ac805c9e0f5486297360ee8588defb4"
format_stamp: "Formatted at 2026-06-20 04:34:42 on dist-test-slave-rzqm"
server_key: "692df8eb5f37de2bbb9ef913005f8941"
server_key_iv: "eed933a4ec8009fdcda4dbf250304734"
server_key_version: "encryptionkey@0"
I20260620 04:34:42.536727 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ac805c9e0f5486297360ee8588defb4"
format_stamp: "Formatted at 2026-06-20 04:34:42 on dist-test-slave-rzqm"
server_key: "692df8eb5f37de2bbb9ef913005f8941"
server_key_iv: "eed933a4ec8009fdcda4dbf250304734"
server_key_version: "encryptionkey@0"
I20260620 04:34:42.540477 22642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:34:42.542989 22657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:42.544315 22642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 04:34:42.544445 22642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "6ac805c9e0f5486297360ee8588defb4"
format_stamp: "Formatted at 2026-06-20 04:34:42 on dist-test-slave-rzqm"
server_key: "692df8eb5f37de2bbb9ef913005f8941"
server_key_iv: "eed933a4ec8009fdcda4dbf250304734"
server_key_version: "encryptionkey@0"
I20260620 04:34:42.544548 22642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:42.553275 22642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:42.555943 22642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:42.556109 22642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:42.562549 22642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41115
I20260620 04:34:42.562789 22709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41115 every 8 connection(s)
I20260620 04:34:42.563488 22642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:42.566252 22710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:42.570016 22710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:34:42.571493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 22642
I20260620 04:34:42.571620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:42.571869 392 external_mini_cluster.cc:1428] Setting key 4307d2c1751df40191b4d3392a75a36b
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 04:34:42 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:42.587579 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.572568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41115 (local address 127.0.0.1:58414)
0620 04:34:42.573154 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.573218 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:42.574136 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:42.574858 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:42.574864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:42.575303 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:42.575690 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:42.575722 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.576856 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.576860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:42.578071 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:42.578083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.578729 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.578734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:42.578876 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.579438 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:42.579458 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:42.582563 (+ 3105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:42.584528 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:42.584536 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:42.584544 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:42.584772 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:42.585160 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:42.585163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:42.585164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:42.585270 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:42.585877 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:42.585883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:42.586009 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 04:34:42.586151 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 04:34:42.586560 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":94,"thread_start_us":132,"threads_started":1}
I20260620 04:34:42.587579 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.572568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.0.98.62:41115)
0620 04:34:42.573165 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:42.573183 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:42.573557 (+ 374us) server_negotiation.cc:407] Connection header received
0620 04:34:42.574443 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:42.574450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:42.574641 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:42.574808 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:42.575855 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.576716 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.578196 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.578565 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.578682 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.582710 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:42.582715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:42.582726 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:42.582752 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:42.584406 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:42.584882 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:42.584896 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:42.584897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:42.585065 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:42.585372 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:42.585375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:42.585377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:42.585771 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:42.585892 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:42.586284 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 04:34:42.586473 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"spinlock_wait_cycles":79232,"thread_start_us":177,"threads_started":1}
I20260620 04:34:42.589951 22710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:34:42.590690 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.590446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45879 (local address 127.0.0.1:41070)
0620 04:34:42.590554 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.590625 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 04:34:42.590915 22660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45879: connect: Connection refused (error 111)
I20260620 04:34:42.591809 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.591569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45879 (local address 127.0.0.1:41084)
0620 04:34:42.591702 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.591751 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 04:34:42.591961 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45879: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45879: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 04:34:42.595309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45879
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41115,127.0.98.61:45879,127.0.98.60:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:34:42.641233 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } attempt: 1
I20260620 04:34:42.642129 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.641829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45879 (local address 127.0.0.1:41088)
0620 04:34:42.641989 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.642051 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 04:34:42.642884 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.642661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45879 (local address 127.0.0.1:41104)
0620 04:34:42.642774 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.642823 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 04:34:42.643038 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45879: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45879: connect: Connection refused (error 111)
W20260620 04:34:42.704962 22717 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:34:42.705279 22717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:42.705302 22717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:42.705339 22717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:42.709307 22717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:42.709375 22717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:42.709393 22717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:42.709408 22717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:42.709422 22717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:42.711246 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } attempt: 2
I20260620 04:34:42.712061 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.711781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45879 (local address 127.0.0.1:41110)
0620 04:34:42.711921 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.711983 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 04:34:42.712898 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.712642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45879 (local address 127.0.0.1:41114)
0620 04:34:42.712774 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.712831 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 04:34:42.713050 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45879: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45879: connect: Connection refused (error 111)
I20260620 04:34:42.715034 22717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41115,127.0.98.61:45879,127.0.98.60:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:42.716051 22717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:42.717072 22717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:42.725785 22722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:42.725767 22725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:42.725771 22723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:42.726073 22717 server_base.cc:1034] running on GCE node
I20260620 04:34:42.726644 22717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:42.727759 22717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:42.728957 22717 hybrid_clock.cc:648] HybridClock initialized: now 1781930082728933 us; error 33 us; skew 500 ppm
Jun 20 04:34:42 dist-test-slave-rzqm krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:42.731904 22717 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 04:34:42.732836 22717 webserver.cc:460] Webserver started at http://127.0.98.61:38287/ using document root <none> and password file <none>
I20260620 04:34:42.733420 22717 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:42.733489 22717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:42.733704 22717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:42.735761 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/data/instance:
uuid: "f7023b8d46004a9c8d1530fb954d8710"
format_stamp: "Formatted at 2026-06-20 04:34:42 on dist-test-slave-rzqm"
server_key: "8223693d87c8d85c6e1d75db1c646160"
server_key_iv: "cfd63aea45c578a57e991a8675fd95d3"
server_key_version: "encryptionkey@0"
I20260620 04:34:42.736315 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/wal/instance:
uuid: "f7023b8d46004a9c8d1530fb954d8710"
format_stamp: "Formatted at 2026-06-20 04:34:42 on dist-test-slave-rzqm"
server_key: "8223693d87c8d85c6e1d75db1c646160"
server_key_iv: "cfd63aea45c578a57e991a8675fd95d3"
server_key_version: "encryptionkey@0"
I20260620 04:34:42.739943 22717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 04:34:42.742558 22732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:42.743773 22717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:42.743894 22717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/wal
uuid: "f7023b8d46004a9c8d1530fb954d8710"
format_stamp: "Formatted at 2026-06-20 04:34:42 on dist-test-slave-rzqm"
server_key: "8223693d87c8d85c6e1d75db1c646160"
server_key_iv: "cfd63aea45c578a57e991a8675fd95d3"
server_key_version: "encryptionkey@0"
I20260620 04:34:42.743991 22717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:42.766702 22717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:42.769937 22717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:42.770124 22717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:42.776722 22717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45879
I20260620 04:34:42.776988 22784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45879 every 8 connection(s)
I20260620 04:34:42.777704 22717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:34:42.780435 22785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:42.781816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 22717
I20260620 04:34:42.781932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-1/wal/instance
I20260620 04:34:42.782181 392 external_mini_cluster.cc:1428] Setting key a8094317ade2f27644375ff1364e4b4a
I20260620 04:34:42.784543 22785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 04:34:42 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:42.798749 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.786563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.0.98.62:41115)
0620 04:34:42.786685 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:42.786688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:42.787773 (+ 1085us) server_negotiation.cc:407] Connection header received
0620 04:34:42.788739 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:42.788742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:42.788806 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:42.788880 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:42.790133 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.790761 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.792154 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.792402 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.792496 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.795301 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:42.795304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:42.795306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:42.795332 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:42.796703 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:42.797240 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:42.797244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:42.797246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:42.797324 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:42.797648 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:42.797652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:42.797653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:42.797833 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:42.797920 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:42.798455 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 04:34:42.798582 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 04:34:42.799582 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.787104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41115 (local address 127.0.0.1:58422)
0620 04:34:42.787616 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.787658 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:42.788522 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:42.789104 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:42.789112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:42.789452 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:42.789987 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:42.790004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.790893 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.790896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:42.792035 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:42.792047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.792549 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.792554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:42.792719 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.793371 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:42.793394 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:42.795127 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:42.796811 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:42.796817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:42.796833 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:42.797148 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:42.797412 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:42.797415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:42.797417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:42.797555 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:42.797931 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:42.797937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:42.798195 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 04:34:42.798358 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 04:34:42.798687 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":142,"threads_started":1}
I20260620 04:34:42.801082 22785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:34:42.804209 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 04:34:42.805913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39585
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41115,127.0.98.61:45879,127.0.98.60:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 04:34:42 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:34:42 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:34:42.810622 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.801558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45879 (local address 127.0.0.1:41124)
0620 04:34:42.801682 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.801703 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:42.801789 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:42.802413 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:42.802415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:42.802544 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:42.802642 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:42.802647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.803658 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.803661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:42.804172 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:42.804182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.805070 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.805073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:42.805171 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.805636 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:42.805649 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:42.806992 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:42.809111 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:42.809114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:42.809116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:42.809331 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:42.809623 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:42.809626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:42.809628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:42.809715 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:42.810171 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:42.810174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:42.810253 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 04:34:42.810340 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 04:34:42.810460 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 04:34:42.810753 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.801651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.0.98.61:45879)
0620 04:34:42.802026 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:42.802040 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:42.802059 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:34:42.802092 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:42.802096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:42.802301 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:42.802416 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:42.802721 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.803561 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.804317 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.804926 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.805369 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.807129 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:42.807138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:42.807154 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:42.807178 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:42.808971 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:42.809446 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:42.809452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:42.809454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:42.809544 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:42.809803 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:42.809806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:42.809808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:42.810075 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:42.810192 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:42.810428 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 04:34:42.810554 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":191,"threads_started":1}
I20260620 04:34:42.813086 22785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:34:42.813750 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.813489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57592)
0620 04:34:42.813591 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.813679 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 04:34:42.813978 22734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
I20260620 04:34:42.814010 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.804619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45879 (local address 127.0.0.1:41126)
0620 04:34:42.804767 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.804782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:42.804860 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:42.805221 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:42.805224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:42.805388 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:42.805493 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:42.805499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.806787 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.806790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:42.807464 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:42.807476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.808029 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.808033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:42.808139 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.808527 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:42.808541 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:42.809803 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:42.812332 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:42.812335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:42.812337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:42.812587 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:42.812895 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:42.812899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:42.812901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:42.813025 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:42.813502 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:42.813505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:42.813616 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 04:34:42.813702 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 04:34:42.813838 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 04:34:42.814652 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.804661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.0.98.61:45879)
0620 04:34:42.804968 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:42.804977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:42.804988 (+ 11us) server_negotiation.cc:407] Connection header received
0620 04:34:42.805039 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:42.805042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:42.805082 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:42.805193 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:42.806081 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.806630 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.807566 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.807911 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.808006 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.809911 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:42.809915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:42.809917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:42.809937 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:42.812214 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:42.812684 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:42.812688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:42.812690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:42.812775 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:42.813149 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:42.813152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:42.813154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:42.813382 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:42.814285 (+ 903us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:42.814377 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 04:34:42.814490 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260620 04:34:42.814957 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.814757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57608)
0620 04:34:42.814846 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.814901 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 04:34:42.815218 22785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
I20260620 04:34:42.815336 22710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:34:42.815963 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.815707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57612)
0620 04:34:42.815832 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.815889 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 04:34:42.816670 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.816406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57628)
0620 04:34:42.816554 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.816605 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 04:34:42.816808 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
I20260620 04:34:42.847949 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } attempt: 1
I20260620 04:34:42.848901 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.848565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57644)
0620 04:34:42.848749 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.848818 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 04:34:42.849627 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.849431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57660)
0620 04:34:42.849524 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.849568 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 04:34:42.849769 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
I20260620 04:34:42.864499 22785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } attempt: 1
I20260620 04:34:42.865411 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.865094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57668)
0620 04:34:42.865264 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.865331 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 04:34:42.866114 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.865908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57684)
0620 04:34:42.866011 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.866057 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 04:34:42.866288 22785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
W20260620 04:34:42.919653 22792 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:34:42.919973 22792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:42.919996 22792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:42.920046 22792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:42.924525 22792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:42.924595 22792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:42.924614 22792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:42.924630 22792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:42.924645 22792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:42.924937 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } attempt: 2
I20260620 04:34:42.925873 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.925542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57696)
0620 04:34:42.925719 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.925790 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 04:34:42.926548 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.926350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57708)
0620 04:34:42.926444 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.926488 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 04:34:42.926682 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
I20260620 04:34:42.930613 22792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41115,127.0.98.61:45879,127.0.98.60:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:42.931699 22792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:42.932701 22792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:34:42.934479 22785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } attempt: 2
I20260620 04:34:42.935235 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.934969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57712)
0620 04:34:42.935103 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.935160 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 04:34:42.936061 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.935819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57720)
0620 04:34:42.935948 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.935999 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 04:34:42.936192 22785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39585: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39585: connect: Connection refused (error 111)
W20260620 04:34:42.941402 22798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:42.941391 22799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:42.941394 22801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:42.941828 22792 server_base.cc:1034] running on GCE node
I20260620 04:34:42.942428 22792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:42.943647 22792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:42.944808 22792 hybrid_clock.cc:648] HybridClock initialized: now 1781930082944778 us; error 47 us; skew 500 ppm
Jun 20 04:34:42 dist-test-slave-rzqm krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:42.947583 22792 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 04:34:42.948585 22792 webserver.cc:460] Webserver started at http://127.0.98.60:38241/ using document root <none> and password file <none>
I20260620 04:34:42.949225 22792 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:42.949295 22792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:42.949499 22792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:42.951589 22792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/data/instance:
uuid: "04a08de940b24c8dbc65644424af445a"
format_stamp: "Formatted at 2026-06-20 04:34:42 on dist-test-slave-rzqm"
server_key: "71bbb83470712c762fbe1e89c869f0b2"
server_key_iv: "634305090b71b3ada4ca91a88df7b76b"
server_key_version: "encryptionkey@0"
I20260620 04:34:42.952153 22792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/wal/instance:
uuid: "04a08de940b24c8dbc65644424af445a"
format_stamp: "Formatted at 2026-06-20 04:34:42 on dist-test-slave-rzqm"
server_key: "71bbb83470712c762fbe1e89c869f0b2"
server_key_iv: "634305090b71b3ada4ca91a88df7b76b"
server_key_version: "encryptionkey@0"
I20260620 04:34:42.956073 22792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:34:42.958583 22808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:42.959880 22792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:42.960001 22792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/wal
uuid: "04a08de940b24c8dbc65644424af445a"
format_stamp: "Formatted at 2026-06-20 04:34:42 on dist-test-slave-rzqm"
server_key: "71bbb83470712c762fbe1e89c869f0b2"
server_key_iv: "634305090b71b3ada4ca91a88df7b76b"
server_key_version: "encryptionkey@0"
I20260620 04:34:42.960101 22792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:42.970764 22792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:42.974159 22792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:42.974345 22792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:42.980867 22792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39585
I20260620 04:34:42.981148 22860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39585 every 8 connection(s)
I20260620 04:34:42.981904 22792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:34:42.983552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 22792
I20260620 04:34:42.983676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781929807654528-392-0/minicluster-data/master-2/wal/instance
I20260620 04:34:42.983902 392 external_mini_cluster.cc:1428] Setting key 5b91921e5a5b065c059434a3e243da98
I20260620 04:34:42.985155 22861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:42.989853 22861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:34:42 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:34:42 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:42.999275 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.985285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.0.98.60:39585)
0620 04:34:42.985710 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:42.985729 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:42.985771 (+ 42us) server_negotiation.cc:407] Connection header received
0620 04:34:42.986620 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:42.986644 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:42.987059 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:42.987448 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:42.988126 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.989659 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.990551 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.991234 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.991384 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.994344 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:42.994349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:42.994360 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:42.994391 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:42.996289 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:42.996723 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:42.996729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:42.996734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:42.996848 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:42.997162 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:42.997165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:42.997166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:42.997688 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:42.997875 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:42.998157 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 04:34:42.998434 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":124,"threads_started":1}
I20260620 04:34:43.001446 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.991761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41115 (local address 127.0.0.1:58436)
0620 04:34:42.992188 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:42.992290 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:42.992422 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:42.992693 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:42.992700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:42.993230 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:42.993382 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:42.993388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.994237 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.994288 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:42.994926 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:42.994935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.995459 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.995461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:42.995548 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.996006 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:42.996018 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:42.997583 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:42.999782 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:42.999789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:42.999794 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:43.000057 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:43.000329 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:43.000332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:43.000334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:43.000426 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:43.000881 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:43.000885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:43.000981 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 04:34:43.001120 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 04:34:43.001264 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":137,"threads_started":1}
I20260620 04:34:43.001557 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:42.991890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.0.98.62:41115)
0620 04:34:42.992045 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:42.992050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:42.992381 (+ 331us) server_negotiation.cc:407] Connection header received
0620 04:34:42.992509 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:42.992511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:42.992554 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:42.992616 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:42.993510 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.994114 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.995049 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:42.995337 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:42.995457 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:42.997748 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:42.997751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:42.997753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:42.997779 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:42.999687 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.000166 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.000169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.000171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.000250 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.000547 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.000551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.000553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.000778 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:43.000882 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:43.001212 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 04:34:43.001385 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:34:43.002684 22861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:34:43 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 04:34:43 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:34:43.010689 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.002103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.0.98.62:41115)
0620 04:34:43.002207 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.002210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.002223 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:34:43.002261 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.002264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.002300 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.002360 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.003336 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.003982 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.004779 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.004986 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.005107 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.007611 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:43.007616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:43.007618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:43.007646 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:43.009189 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.009566 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.009570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.009572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.009660 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.009901 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.009904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.009906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.010083 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:43.010291 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:43.010424 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 04:34:43.010531 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 04:34:43.010900 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.003193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45879 (local address 127.0.0.1:41140)
0620 04:34:43.003425 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:43.003436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:43.003485 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:43.003677 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:43.003679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:43.003797 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:43.003902 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:43.003907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.004628 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.004631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:43.005293 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:43.005304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.005890 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.005894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:43.005992 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.006345 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:43.006359 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:43.007829 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:43.009569 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:43.009572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:43.009574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:43.009769 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:43.010013 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:43.010016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:43.010018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:43.010090 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:43.010430 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:43.010432 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:43.010520 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 04:34:43.010602 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 04:34:43.010736 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260620 04:34:43.011032 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.003311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.0.98.61:45879)
0620 04:34:43.003440 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.003444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.003476 (+ 32us) server_negotiation.cc:407] Connection header received
0620 04:34:43.003550 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.003553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.003595 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.003671 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.003994 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.004528 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.005407 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.005767 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.005847 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.008025 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:43.008029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:43.008033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:43.008064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:43.009464 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.009856 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.009859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.009861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.009934 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.010165 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.010176 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.010178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.010343 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:43.010423 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:43.010659 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 04:34:43.010892 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 04:34:43.011619 22861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:34:43 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:34:43 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:34:43.019622 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.012046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.0.98.61:45879)
0620 04:34:43.012147 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.012151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.012163 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:34:43.012196 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.012199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.012241 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.012313 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.012959 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.013548 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.014269 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.014480 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.014565 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.016852 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:43.016855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:43.016857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:43.016876 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:43.018101 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.018510 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.018514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.018516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.018592 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.018830 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.018833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.018835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.019036 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:43.019182 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:43.019357 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 04:34:43.019470 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 04:34:43.019799 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.012178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.0.98.60:39585)
0620 04:34:43.012283 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.012287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.012300 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:34:43.012379 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.012381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.012422 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.012503 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.012854 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.013433 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.014166 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.014464 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.014543 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.016276 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:43.016279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:43.016282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:43.016304 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:43.018240 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.018611 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.018613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.018615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.018698 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.018959 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.018961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.018963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.019142 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:43.019233 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:43.019541 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 04:34:43.019649 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 04:34:43.019907 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.012073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57724)
0620 04:34:43.012223 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:43.012235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:43.012298 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:43.012545 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:43.012547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:43.012671 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:43.012747 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:43.012752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.013528 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.013532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:43.014058 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:43.014066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.014561 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.014565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:43.014636 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.015043 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:43.015057 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:43.016165 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:43.018336 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:43.018338 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:43.018339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:43.018523 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:43.018787 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:43.018789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:43.018791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:43.018880 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:43.019243 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:43.019246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:43.019352 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 04:34:43.019454 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 04:34:43.019744 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":27}
I20260620 04:34:43.023191 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a: Bootstrap starting.
I20260620 04:34:43.023453 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:43.024814 22684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6ac805c9e0f5486297360ee8588defb4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58446
I20260620 04:34:43.025852 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } attempt: 3
I20260620 04:34:43.025903 22861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:43.026818 22861 log.cc:824] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:43.027324 22785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } attempt: 3
I20260620 04:34:43.029407 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a: No bootstrap required, opened a new log
Jun 20 04:34:43 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:34:43 dist-test-slave-rzqm krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:34:43.033579 22861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } }
I20260620 04:34:43.033854 22861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a08de940b24c8dbc65644424af445a, State: Initialized, Role: FOLLOWER
I20260620 04:34:43.034219 22861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } }
I20260620 04:34:43.034857 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.026277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57732)
0620 04:34:43.026410 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:43.026426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:43.026519 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:43.026769 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:43.026772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:43.026964 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:43.027075 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:43.027082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.028357 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.028361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:43.028885 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:43.028893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.029429 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.029432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:43.029510 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.029988 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:43.030002 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:43.031491 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:43.033279 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:43.033282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:43.033284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:43.033529 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:43.033829 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:43.033832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:43.033834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:43.033930 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:43.034349 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:43.034352 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:43.034448 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 04:34:43.034542 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 04:34:43.034681 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 04:34:43.034796 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } } }
I20260620 04:34:43.035099 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:43.035218 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.026323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.0.98.60:39585)
0620 04:34:43.026434 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.026438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.026470 (+ 32us) server_negotiation.cc:407] Connection header received
0620 04:34:43.026605 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.026607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.026646 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.026729 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.027179 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.028269 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.028989 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.029326 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.029417 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.031659 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:43.031663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:43.031666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:43.031696 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:43.033159 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.033614 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.033618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.033619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.033724 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.034046 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.034049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.034051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.034256 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:43.034344 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:43.034614 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 04:34:43.034727 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:34:43.035667 22861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:43.037530 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.027715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39585 (local address 127.0.0.1:57738)
0620 04:34:43.027830 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:43.027846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:43.027929 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:43.028289 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:43.028292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:43.028471 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:43.028578 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:43.028585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.029702 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.029706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:43.030526 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:43.030538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.031173 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.031177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:43.031258 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.031730 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:43.031744 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:43.033139 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:43.035963 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:43.035966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:43.035968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:43.036230 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:43.036542 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:43.036545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:43.036547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:43.036639 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:43.037023 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:43.037027 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:43.037122 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 04:34:43.037227 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 04:34:43.037374 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 04:34:43.038149 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.027773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.0.98.60:39585)
0620 04:34:43.028064 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.028072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.028085 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:34:43.028116 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.028119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.028155 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.028681 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.028784 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.029587 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.030652 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.031036 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.031162 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.033297 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:43.033318 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:43.033321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:43.033345 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:43.035728 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.036334 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.036338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.036340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.036420 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.036714 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.036717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.036719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.036917 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:43.037018 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:43.037865 (+ 847us) server_negotiation.cc:299] Negotiation successful
0620 04:34:43.037977 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
I20260620 04:34:43.038412 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4: Bootstrap starting.
W20260620 04:34:43.040601 22879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:43.040676 22879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:43.040947 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710: Bootstrap starting.
I20260620 04:34:43.041103 22710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:43.041940 22710 log.cc:824] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:43.043431 22785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:43.044229 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4: No bootstrap required, opened a new log
I20260620 04:34:43.044237 22785 log.cc:824] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:43.043941 22792 master_runner.cc:186] Error getting master registration for 127.0.98.62:41115: OK, instance_id { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" instance_seqno: 1781930082560805 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:34:43.047201 22792 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:34:43.047150 22710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } }
I20260620 04:34:43.047369 22710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac805c9e0f5486297360ee8588defb4, State: Initialized, Role: FOLLOWER
I20260620 04:34:43.047705 22710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } }
I20260620 04:34:43.048247 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } } }
I20260620 04:34:43.048444 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:43.048831 22710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:43.048949 22684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 04:34:43.049105 22684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:43.050335 22684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } }
I20260620 04:34:43.051301 22684 leader_election.cc:288] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [CANDIDATE]: Term 1 election: Requested vote from peers f7023b8d46004a9c8d1530fb954d8710 (127.0.98.61:45879), 04a08de940b24c8dbc65644424af445a (127.0.98.60:39585)
I20260620 04:34:43.052130 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ac805c9e0f5486297360ee8588defb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "04a08de940b24c8dbc65644424af445a"
I20260620 04:34:43.052356 22835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:43.054142 22835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ac805c9e0f5486297360ee8588defb4 in term 1.
W20260620 04:34:43.054154 22894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:43.054219 22894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:43.054570 22659 leader_election.cc:302] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a08de940b24c8dbc65644424af445a, 6ac805c9e0f5486297360ee8588defb4; no voters:
I20260620 04:34:43.054833 22883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:43.055022 22883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [term 1 LEADER]: Becoming Leader. State: Replica: 6ac805c9e0f5486297360ee8588defb4, State: Running, Role: LEADER
I20260620 04:34:43.055341 22883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } }
W20260620 04:34:43.057624 22663 tablet.cc:2369] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:34:43.057745 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac805c9e0f5486297360ee8588defb4. Latest consensus state: current_term: 1 leader_uuid: "6ac805c9e0f5486297360ee8588defb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } } }
I20260620 04:34:43.057899 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:43.061622 22642 master_runner.cc:186] Error getting master registration for 127.0.98.61:45879: OK, instance_id { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" instance_seqno: 1781930082774989 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:34:43.064924 22642 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:34:43.066958 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710: No bootstrap required, opened a new log
I20260620 04:34:43.068691 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.057484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57752 (local address 127.0.98.60:39585)
0620 04:34:43.058088 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.058093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.058689 (+ 596us) server_negotiation.cc:407] Connection header received
0620 04:34:43.058884 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.058888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.058934 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.058983 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.059957 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.060759 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.061919 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.062265 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.062336 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.064402 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:43.064406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:43.064409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:43.064438 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:43.066269 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.066966 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.066969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.066971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.067059 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.067493 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.067496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.067498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.067731 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:43.067840 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:43.068408 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 04:34:43.068537 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520}
I20260620 04:34:43.071254 22785 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } }
I20260620 04:34:43.071520 22785 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7023b8d46004a9c8d1530fb954d8710, State: Initialized, Role: FOLLOWER
I20260620 04:34:43.071892 22785 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } }
I20260620 04:34:43.073395 22785 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:43.073818 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.059078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.0.98.61:45879)
0620 04:34:43.059185 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.059190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.059732 (+ 542us) server_negotiation.cc:407] Connection header received
0620 04:34:43.059774 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.059776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.059820 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.059905 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.061395 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.062122 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.066565 (+ 4443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.066912 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.066983 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.069687 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:43.069691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:43.069695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:43.069724 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:43.071264 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.072411 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.072415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.072417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.072507 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.072804 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.072807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.072809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.073080 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:43.073195 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:43.073547 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 04:34:43.073660 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 04:34:43.074286 22759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ac805c9e0f5486297360ee8588defb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7023b8d46004a9c8d1530fb954d8710"
I20260620 04:34:43.074540 22759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:43.075947 22759 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ac805c9e0f5486297360ee8588defb4 in term 1.
I20260620 04:34:43.076828 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:43.058746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.0.98.62:41115)
0620 04:34:43.058844 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.058848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.059365 (+ 517us) server_negotiation.cc:407] Connection header received
0620 04:34:43.059510 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.059513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.059574 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.059651 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.064566 (+ 4915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.065319 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.068578 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:43.068890 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:43.068965 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:43.072087 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:43.072091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:43.072094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:43.072125 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:43.073941 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.075444 (+ 1503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.075448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.075449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.075545 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:43.075852 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:43.075855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:43.075869 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:43.076091 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:43.076217 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:43.076559 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 04:34:43.076677 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 04:34:43.076848 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } } }
I20260620 04:34:43.081045 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 04:34:43.084165 22917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:43.084246 22917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:43.084537 22902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:43.085456 22902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:43.093237 22835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [term 1 FOLLOWER]: Refusing update from remote peer 6ac805c9e0f5486297360ee8588defb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:43.093284 22759 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [term 1 FOLLOWER]: Refusing update from remote peer 6ac805c9e0f5486297360ee8588defb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:43.093875 22900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:43.094291 22900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:43.098240 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac805c9e0f5486297360ee8588defb4. Latest consensus state: current_term: 1 leader_uuid: "6ac805c9e0f5486297360ee8588defb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } } }
I20260620 04:34:43.098240 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ac805c9e0f5486297360ee8588defb4. Latest consensus state: current_term: 1 leader_uuid: "6ac805c9e0f5486297360ee8588defb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } } }
I20260620 04:34:43.098356 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:43.098356 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:43.110157 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ac805c9e0f5486297360ee8588defb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } } }
I20260620 04:34:43.110294 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:43.111192 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ac805c9e0f5486297360ee8588defb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } } }
I20260620 04:34:43.111317 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:43.111651 22919 mvcc.cc:205] Tried to move back new op lower bound from 7298785620343005184 to 7298785620197892096. Current Snapshot: MvccSnapshot[applied={T|T < 7298785620343005184}]
I20260620 04:34:43.112232 22921 mvcc.cc:205] Tried to move back new op lower bound from 7298785620343005184 to 7298785620197892096. Current Snapshot: MvccSnapshot[applied={T|T < 7298785620343005184}]
I20260620 04:34:43.112673 22902 catalog_manager.cc:1347] Generated new cluster ID: 65319763d5af4df6819cc7a10fd858f0
I20260620 04:34:43.112746 22902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:43.113082 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ac805c9e0f5486297360ee8588defb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } } }
I20260620 04:34:43.113216 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4 [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:43.114244 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ac805c9e0f5486297360ee8588defb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac805c9e0f5486297360ee8588defb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } peers { permanent_uuid: "f7023b8d46004a9c8d1530fb954d8710" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45879 } } peers { permanent_uuid: "04a08de940b24c8dbc65644424af445a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39585 } } }
I20260620 04:34:43.114365 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:43.114928 22920 mvcc.cc:205] Tried to move back new op lower bound from 7298785620343005184 to 7298785620197892096. Current Snapshot: MvccSnapshot[applied={T|T < 7298785620343005184}]
I20260620 04:34:43.128614 22902 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:43.129931 22902 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:43.140966 22902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ac805c9e0f5486297360ee8588defb4: Generated new TSK 0
I20260620 04:34:43.141883 22902 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:34:43.187929 22867 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.185143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.98.60:39585)
0620 04:34:43.185280 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.185285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.186765 (+ 1480us) server_negotiation.cc:407] Connection header received
0620 04:34:43.186985 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.186989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.187038 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.187090 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.187767 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57760: BlockingRecv error: recv got EOF from 127.0.0.1:57760 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 04:34:43.189826 22713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.184623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.0.98.62:41115)
0620 04:34:43.184779 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.184785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.188602 (+ 3817us) server_negotiation.cc:407] Connection header received
0620 04:34:43.188958 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.188963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.189040 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.189094 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.189692 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58454: BlockingRecv error: recv got EOF from 127.0.0.1:58454 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 04:34:43.191388 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.186123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.0.98.61:45879)
0620 04:34:43.186233 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.186238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.190454 (+ 4216us) server_negotiation.cc:407] Connection header received
0620 04:34:43.190641 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.190644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.190700 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.190752 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.191266 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41152: BlockingRecv error: recv got EOF from 127.0.0.1:41152 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 04:34:43.196303 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.194698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.0.98.61:45879)
0620 04:34:43.194821 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.194825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.195312 (+ 487us) server_negotiation.cc:407] Connection header received
0620 04:34:43.195521 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.195525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.195600 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.195653 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.196198 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41166: BlockingRecv error: recv got EOF from 127.0.0.1:41166 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 04:34:43.201651 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.199989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.0.98.61:45879)
0620 04:34:43.200110 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.200121 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.200694 (+ 573us) server_negotiation.cc:407] Connection header received
0620 04:34:43.200896 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.200899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.200957 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.201036 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.201546 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41178: BlockingRecv error: recv got EOF from 127.0.0.1:41178 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 04:34:43.207579 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.206040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.0.98.61:45879)
0620 04:34:43.206144 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.206148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.206655 (+ 507us) server_negotiation.cc:407] Connection header received
0620 04:34:43.206852 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.206856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.206905 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.206953 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.207461 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41192: BlockingRecv error: recv got EOF from 127.0.0.1:41192 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 04:34:43.215816 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.213978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.0.98.61:45879)
0620 04:34:43.214125 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.214130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.214748 (+ 618us) server_negotiation.cc:407] Connection header received
0620 04:34:43.214967 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.214971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.215029 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.215084 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.215674 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41196: BlockingRecv error: recv got EOF from 127.0.0.1:41196 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 04:34:43.229493 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.227543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.0.98.61:45879)
0620 04:34:43.227672 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.227677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.228356 (+ 679us) server_negotiation.cc:407] Connection header received
0620 04:34:43.228580 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.228584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.228641 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.228698 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.229366 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41202: BlockingRecv error: recv got EOF from 127.0.0.1:41202 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 04:34:43.250216 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.248179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.0.98.61:45879)
0620 04:34:43.248324 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.248329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.248962 (+ 633us) server_negotiation.cc:407] Connection header received
0620 04:34:43.249304 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.249308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.249365 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.249417 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.250059 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41204: BlockingRecv error: recv got EOF from 127.0.0.1:41204 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 04:34:43.285943 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.284689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.0.98.61:45879)
0620 04:34:43.284849 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.284854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.285094 (+ 240us) server_negotiation.cc:407] Connection header received
0620 04:34:43.285233 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.285237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.285291 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.285359 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.285821 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41208: BlockingRecv error: recv got EOF from 127.0.0.1:41208 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 04:34:43.353111 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.351990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.0.98.61:45879)
0620 04:34:43.352164 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.352170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.352259 (+ 89us) server_negotiation.cc:407] Connection header received
0620 04:34:43.352387 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.352390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.352440 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.352512 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.352966 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41222: BlockingRecv error: recv got EOF from 127.0.0.1:41222 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 04:34:43.484164 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.483112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.0.98.61:45879)
0620 04:34:43.483274 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.483279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.483412 (+ 133us) server_negotiation.cc:407] Connection header received
0620 04:34:43.483539 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.483542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.483593 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.483653 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.484044 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41224: BlockingRecv error: recv got EOF from 127.0.0.1:41224 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 04:34:43.743422 22791 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:43.742355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.0.98.61:45879)
0620 04:34:43.742480 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:43.742485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:43.742542 (+ 57us) server_negotiation.cc:407] Connection header received
0620 04:34:43.742699 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:43.742702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:43.742748 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:43.742817 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:43.743302 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41240: BlockingRecv error: recv got EOF from 127.0.0.1:41240 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 04:34:44.042106 22879 catalog_manager.cc:1259] Loaded cluster ID: 65319763d5af4df6819cc7a10fd858f0
I20260620 04:34:44.042212 22879 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a: loading cluster ID for follower catalog manager: success
I20260620 04:34:44.043948 22879 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a: acquiring CA information for follower catalog manager: success
I20260620 04:34:44.045272 22879 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04a08de940b24c8dbc65644424af445a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:34:44.085556 22917 catalog_manager.cc:1259] Loaded cluster ID: 65319763d5af4df6819cc7a10fd858f0
I20260620 04:34:44.085644 22917 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710: loading cluster ID for follower catalog manager: success
I20260620 04:34:44.087342 22917 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710: acquiring CA information for follower catalog manager: success
I20260620 04:34:44.088472 22917 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7023b8d46004a9c8d1530fb954d8710: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:34:44.260936 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:44.257418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.0.98.61:45879)
0620 04:34:44.257637 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:44.257641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:44.257657 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:34:44.257758 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:44.257761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:44.257884 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:44.257978 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:44.258218 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:44.258750 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:44.259564 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:44.259793 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:44.259868 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:44.260130 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:44.260562 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:44.260658 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:44.260687 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 04:34:44.260738 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260620 04:34:44.267517 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:44.263861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.0.98.60:39585)
0620 04:34:44.264164 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:44.264168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:44.264190 (+ 22us) server_negotiation.cc:407] Connection header received
0620 04:34:44.264270 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:44.264272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:44.264382 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:44.264482 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:44.264668 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:44.265228 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:44.266055 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:44.266290 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:44.266373 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:44.266607 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:44.267115 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:44.267211 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:44.267276 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 04:34:44.267326 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
I20260620 04:34:44.274228 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:44.270472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.0.98.60:39585)
0620 04:34:44.270584 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:44.270588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:44.270714 (+ 126us) server_negotiation.cc:407] Connection header received
0620 04:34:44.270876 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:44.270879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:44.270920 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:44.270976 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:44.271192 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:44.271862 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:44.272981 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:44.273347 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:44.273450 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:44.273754 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:44.273835 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:44.273879 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:44.274039 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 04:34:44.274092 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 04:34:44.274672 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:44.270357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.0.98.62:41115)
0620 04:34:44.270711 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:44.270715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:44.270729 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:34:44.270880 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:44.270883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:44.271049 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:44.271169 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:44.271365 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:44.272086 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:44.273160 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:44.273381 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:44.273470 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:44.273861 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:44.274242 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:44.274297 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:44.274458 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 04:34:44.274517 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":65,"threads_started":1}
I20260620 04:34:44.274750 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:44.270631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.0.98.61:45879)
0620 04:34:44.270739 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:44.270743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:44.271493 (+ 750us) server_negotiation.cc:407] Connection header received
0620 04:34:44.271589 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:44.271592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:44.271645 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:44.271715 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:44.271930 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:44.272495 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:44.273500 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:44.273794 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:44.273894 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:44.274272 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:44.274351 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:44.274399 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:44.274557 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 04:34:44.274627 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 04:34:44.277988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 22642
I20260620 04:34:44.283501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 22717
I20260620 04:34:44.288477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 22792
2026-06-20T04:34:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2961 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 04:34:44 dist-test-slave-rzqm krb5kdc[23013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 04:34:44 dist-test-slave-rzqm krb5kdc[23013](info): set up 2 sockets
Jun 20 04:34:44 dist-test-slave-rzqm krb5kdc[23013](info): commencing operation
krb5kdc: starting...
W20260620 04:34:45.241281 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/test-user.keytab.
Jun 20 04:34:45 dist-test-slave-rzqm krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T04:34:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T04:34:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 04:34:45.356299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41965
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41965,127.0.98.61:38985,127.0.98.60:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 04:34:45.463845 23026 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:34:45.464138 23026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:45.464159 23026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:45.464196 23026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:45.468154 23026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:45.468214 23026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:45.468233 23026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:45.468247 23026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:45.468261 23026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:45.473515 23026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41965,127.0.98.61:38985,127.0.98.60:35165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:45.474505 23026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:45.475457 23026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:45.484047 23032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:45.484107 23026 server_base.cc:1034] running on GCE node
W20260620 04:34:45.484040 23034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:45.484040 23031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:45.484792 23026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:45.485942 23026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:45.487118 23026 hybrid_clock.cc:648] HybridClock initialized: now 1781930085487097 us; error 33 us; skew 500 ppm
Jun 20 04:34:45 dist-test-slave-rzqm krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:45.489989 23026 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 04:34:45.490896 23026 webserver.cc:460] Webserver started at http://127.0.98.62:41649/ using document root <none> and password file <none>
I20260620 04:34:45.491444 23026 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:45.491518 23026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:45.491717 23026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:45.493808 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/data/instance:
uuid: "af7b253ed87f442a9d99cec4674a800f"
format_stamp: "Formatted at 2026-06-20 04:34:45 on dist-test-slave-rzqm"
server_key: "32429af000b459484ed80d0fa5b28e00"
server_key_iv: "ef4dbee0865b3e7f4163be463bea7265"
server_key_version: "encryptionkey@0"
I20260620 04:34:45.494334 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/wal/instance:
uuid: "af7b253ed87f442a9d99cec4674a800f"
format_stamp: "Formatted at 2026-06-20 04:34:45 on dist-test-slave-rzqm"
server_key: "32429af000b459484ed80d0fa5b28e00"
server_key_iv: "ef4dbee0865b3e7f4163be463bea7265"
server_key_version: "encryptionkey@0"
I20260620 04:34:45.498039 23026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:45.500491 23041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:45.501828 23026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 04:34:45.501946 23026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/wal
uuid: "af7b253ed87f442a9d99cec4674a800f"
format_stamp: "Formatted at 2026-06-20 04:34:45 on dist-test-slave-rzqm"
server_key: "32429af000b459484ed80d0fa5b28e00"
server_key_iv: "ef4dbee0865b3e7f4163be463bea7265"
server_key_version: "encryptionkey@0"
I20260620 04:34:45.502063 23026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:45.556609 23026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:45.560003 23026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:45.560210 23026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:45.566869 23026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41965
I20260620 04:34:45.567104 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41965 every 8 connection(s)
I20260620 04:34:45.567835 23026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/data/info.pb
I20260620 04:34:45.570529 23094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:45.572979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 23026
I20260620 04:34:45.573128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-0/wal/instance
I20260620 04:34:45.573439 392 external_mini_cluster.cc:1428] Setting key 1868b0da2a9e736264f227258f98a42a
I20260620 04:34:45.574659 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 04:34:45 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:45.589190 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.577059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.0.98.62:41965)
0620 04:34:45.577618 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:45.577635 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:45.577814 (+ 179us) server_negotiation.cc:407] Connection header received
0620 04:34:45.578789 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:45.578793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:45.578976 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:45.579130 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:45.579916 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.580768 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.581775 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.582130 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.582276 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:45.584731 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:45.584737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:45.584746 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:45.584767 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:45.586263 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:45.586701 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:45.586706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:45.586707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:45.586805 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:45.587070 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:45.587074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:45.587076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:45.587437 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:45.587552 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:45.587891 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 04:34:45.588111 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":437,"thread_start_us":183,"threads_started":1}
I20260620 04:34:45.589197 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.577059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41965 (local address 127.0.0.1:36660)
0620 04:34:45.577673 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.577713 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:45.578504 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:45.579142 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:45.579147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:45.579551 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:45.579786 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:45.579799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.580880 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.580882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:45.581643 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:45.581684 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.582263 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.582267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:45.582375 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:45.582861 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:45.582877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:45.584595 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:45.586370 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:45.586379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:45.586389 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:45.586600 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:45.586885 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:45.586888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:45.586890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:45.586995 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:45.587538 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:45.587544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:45.587671 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 04:34:45.587817 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 04:34:45.588172 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":89,"spinlock_wait_cycles":49408,"thread_start_us":260,"threads_started":1}
I20260620 04:34:45.591922 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:34:45.592712 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.592405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38985 (local address 127.0.0.1:41950)
0620 04:34:45.592527 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.592617 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 04:34:45.592948 23043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38985: connect: Connection refused (error 111)
I20260620 04:34:45.593851 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.593619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38985 (local address 127.0.0.1:41958)
0620 04:34:45.593743 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.593791 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 04:34:45.593989 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38985: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38985: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 04:34:45.599414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38985
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41965,127.0.98.61:38985,127.0.98.60:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:34:45.643262 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } attempt: 1
I20260620 04:34:45.644148 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.643794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38985 (local address 127.0.0.1:41962)
0620 04:34:45.643959 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.644036 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 04:34:45.644919 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.644697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38985 (local address 127.0.0.1:41966)
0620 04:34:45.644801 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.644854 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 04:34:45.645083 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38985: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38985: connect: Connection refused (error 111)
W20260620 04:34:45.708729 23101 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:34:45.709055 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:45.709090 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:45.709133 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 04:34:45.713132 23101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:45.713197 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:45.713225 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:45.713246 23101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:45.713266 23101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:45.713243 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } attempt: 2
I20260620 04:34:45.714049 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.713728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38985 (local address 127.0.0.1:41982)
0620 04:34:45.713890 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.713961 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 04:34:45.714774 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.714545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38985 (local address 127.0.0.1:41994)
0620 04:34:45.714641 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.714703 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 04:34:45.714937 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38985: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38985: connect: Connection refused (error 111)
I20260620 04:34:45.719064 23101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41965,127.0.98.61:38985,127.0.98.60:35165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:45.720146 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:45.721205 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 04:34:45.729429 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:45.729349 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:45.729364 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:45.729609 23101 server_base.cc:1034] running on GCE node
I20260620 04:34:45.730268 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:45.731377 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:45.732553 23101 hybrid_clock.cc:648] HybridClock initialized: now 1781930085732524 us; error 48 us; skew 500 ppm
Jun 20 04:34:45 dist-test-slave-rzqm krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:45.735467 23101 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 04:34:45.736352 23101 webserver.cc:460] Webserver started at http://127.0.98.61:41701/ using document root <none> and password file <none>
I20260620 04:34:45.736883 23101 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:45.736948 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:45.737171 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:45.739192 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/data/instance:
uuid: "dfa8dc6275114840b5f1b9545f26259f"
format_stamp: "Formatted at 2026-06-20 04:34:45 on dist-test-slave-rzqm"
server_key: "a70225020a059c9cdeeda44033af4c51"
server_key_iv: "18edfe8cd1f65d8152ebecae8a334c7a"
server_key_version: "encryptionkey@0"
I20260620 04:34:45.739734 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/wal/instance:
uuid: "dfa8dc6275114840b5f1b9545f26259f"
format_stamp: "Formatted at 2026-06-20 04:34:45 on dist-test-slave-rzqm"
server_key: "a70225020a059c9cdeeda44033af4c51"
server_key_iv: "18edfe8cd1f65d8152ebecae8a334c7a"
server_key_version: "encryptionkey@0"
I20260620 04:34:45.743525 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 04:34:45.745976 23116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:45.747478 23101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 04:34:45.747637 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/wal
uuid: "dfa8dc6275114840b5f1b9545f26259f"
format_stamp: "Formatted at 2026-06-20 04:34:45 on dist-test-slave-rzqm"
server_key: "a70225020a059c9cdeeda44033af4c51"
server_key_iv: "18edfe8cd1f65d8152ebecae8a334c7a"
server_key_version: "encryptionkey@0"
I20260620 04:34:45.747763 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:45.768397 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:45.771725 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:45.771890 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:45.778620 23101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38985
I20260620 04:34:45.778916 23168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38985 every 8 connection(s)
I20260620 04:34:45.779568 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/data/info.pb
I20260620 04:34:45.782437 23169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:45.785786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 23101
I20260620 04:34:45.785925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-1/wal/instance
I20260620 04:34:45.786219 392 external_mini_cluster.cc:1428] Setting key 8d280f28202fb6b6f4c78e6a1985667b
I20260620 04:34:45.786477 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 04:34:45 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:45.800741 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.788790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.0.98.62:41965)
0620 04:34:45.788916 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:45.788919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:45.790127 (+ 1208us) server_negotiation.cc:407] Connection header received
0620 04:34:45.791112 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:45.791116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:45.791196 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:45.791266 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:45.792446 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.792988 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.794166 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.794411 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.794503 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:45.797063 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:45.797067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:45.797070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:45.797099 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:45.798452 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:45.799022 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:45.799025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:45.799027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:45.799124 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:45.799520 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:45.799524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:45.799525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:45.799774 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:45.799877 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:45.800427 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 04:34:45.800563 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:34:45.801528 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.789290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41965 (local address 127.0.0.1:36666)
0620 04:34:45.789966 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.790024 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:45.790902 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:45.791450 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:45.791457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:45.791830 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:45.792300 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:45.792313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.793124 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.793127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:45.794041 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:45.794051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.794557 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.794561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:45.794729 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:45.795283 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:45.795300 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:45.796860 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:45.798591 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:45.798598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:45.798613 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:45.798908 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:45.799278 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:45.799280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:45.799282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:45.799429 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:45.799876 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:45.799882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:45.800132 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 04:34:45.800334 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 04:34:45.800609 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":141,"threads_started":1}
I20260620 04:34:45.803027 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:34:45.806113 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } attempt: 3
Jun 20 04:34:45 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:34:45 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 04:34:45.812297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
/tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35165
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41965,127.0.98.61:38985,127.0.98.60:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 04:34:45.813333 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.803565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38985 (local address 127.0.0.1:42006)
0620 04:34:45.803741 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.803754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:45.803828 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:45.804265 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:45.804267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:45.804460 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:45.804554 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:45.804559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.805689 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.805692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:45.806421 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:45.806430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.807167 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.807171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:45.807296 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:45.807790 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:45.807804 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:45.809061 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:45.811251 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:45.811255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:45.811257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:45.811481 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:45.811944 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:45.811948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:45.811950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:45.812025 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:45.812832 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:45.812835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:45.812924 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 04:34:45.813042 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 04:34:45.813158 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":18}
I20260620 04:34:45.814127 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.803636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.0.98.61:38985)
0620 04:34:45.803905 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:45.803921 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:45.803940 (+ 19us) server_negotiation.cc:407] Connection header received
0620 04:34:45.803974 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:45.803978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:45.804126 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:45.804236 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:45.804650 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.805573 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.806539 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.807067 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.807172 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:45.809213 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:45.809222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:45.809232 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:45.809252 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:45.811109 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:45.811753 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:45.811760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:45.811761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:45.811860 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:45.812095 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:45.812098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:45.812100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:45.812647 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:45.812794 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:45.813790 (+ 996us) server_negotiation.cc:299] Negotiation successful
0620 04:34:45.813925 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260620 04:34:45.815436 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.806587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38985 (local address 127.0.0.1:42016)
0620 04:34:45.806687 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.806700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:45.806772 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:45.807202 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:45.807206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:45.807377 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:45.807477 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:45.807484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.808327 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.808331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:45.808961 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:45.808970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.809490 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.809494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:45.809582 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:45.809920 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:45.809930 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:45.811400 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:45.813982 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:45.813985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:45.813988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:45.814233 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:45.814538 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:45.814542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:45.814544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:45.814626 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:45.814959 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:45.814962 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:45.815055 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 04:34:45.815138 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 04:34:45.815275 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 04:34:45.815452 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.806649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.0.98.61:38985)
0620 04:34:45.807013 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:45.807019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:45.807029 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:34:45.807054 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:45.807056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:45.807087 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:45.807159 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:45.807567 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.808209 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.809105 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.809387 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.809466 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:45.811565 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:45.811569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:45.811572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:45.811595 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:45.813859 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:45.814355 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:45.814358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:45.814359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:45.814433 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:45.814709 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:45.814712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:45.814714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:45.814872 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:45.814952 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:45.815204 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 04:34:45.815301 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":308,"thread_start_us":180,"threads_started":1}
I20260620 04:34:45.816099 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:34:45.816222 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:34:45.817237 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.816576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47656)
0620 04:34:45.817120 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.817184 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469}
I20260620 04:34:45.817534 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.816613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47658)
0620 04:34:45.817394 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.817473 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":717}
W20260620 04:34:45.817775 23118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
I20260620 04:34:45.818065 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.817818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47672)
0620 04:34:45.817979 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.818016 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260620 04:34:45.818222 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35165: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
I20260620 04:34:45.818746 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.818589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47678)
0620 04:34:45.818661 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.818701 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 04:34:45.818893 23169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35165: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
I20260620 04:34:45.849391 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } attempt: 1
I20260620 04:34:45.850178 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.849905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47692)
0620 04:34:45.850038 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.850100 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 04:34:45.850924 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.850714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47700)
0620 04:34:45.850822 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.850868 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 04:34:45.851042 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35165: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
I20260620 04:34:45.868166 23169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } attempt: 1
I20260620 04:34:45.869056 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.868719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47708)
0620 04:34:45.868880 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.868941 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 04:34:45.869711 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.869505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47718)
0620 04:34:45.869604 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.869652 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 04:34:45.869855 23169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35165: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
W20260620 04:34:45.924403 23176 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 04:34:45.924713 23176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 04:34:45.924748 23176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 04:34:45.924810 23176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 04:34:45.926203 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } attempt: 2
I20260620 04:34:45.927016 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.926719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47730)
0620 04:34:45.926872 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.926937 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 04:34:45.927769 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.927543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47744)
0620 04:34:45.927650 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.927711 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 04:34:45.927873 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35165: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
W20260620 04:34:45.929201 23176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 04:34:45.929250 23176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 04:34:45.929277 23176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 04:34:45.929308 23176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 04:34:45.929337 23176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 04:34:45.934748 23176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41965,127.0.98.61:38985,127.0.98.60:35165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 03:49:13 UTC on ccycloud.quasar-kghopn.root.comops.site
build id 4456962
I20260620 04:34:45.935828 23176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 04:34:45.936880 23176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 04:34:45.938006 23169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } attempt: 2
I20260620 04:34:45.938828 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.938524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47756)
0620 04:34:45.938682 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.938743 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 04:34:45.939545 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.939341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47766)
0620 04:34:45.939441 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.939486 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 04:34:45.939683 23169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35165: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35165: connect: Connection refused (error 111)
W20260620 04:34:45.945713 23183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:45.945719 23185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 04:34:45.945719 23182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 04:34:45.946048 23176 server_base.cc:1034] running on GCE node
I20260620 04:34:45.946643 23176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 04:34:45.947707 23176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 04:34:45.948866 23176 hybrid_clock.cc:648] HybridClock initialized: now 1781930085948849 us; error 30 us; skew 500 ppm
Jun 20 04:34:45 dist-test-slave-rzqm krb5kdc[23013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 04:34:45.951570 23176 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 04:34:45.952461 23176 webserver.cc:460] Webserver started at http://127.0.98.60:45075/ using document root <none> and password file <none>
I20260620 04:34:45.953033 23176 fs_manager.cc:359] Metadata directory not provided
I20260620 04:34:45.953112 23176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 04:34:45.953291 23176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 04:34:45.955402 23176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/data/instance:
uuid: "c3a065980def4205908e29c20735648c"
format_stamp: "Formatted at 2026-06-20 04:34:45 on dist-test-slave-rzqm"
server_key: "f409e00e38d1238a52f4816859994a9c"
server_key_iv: "904c73ebf1dd4a395b237664f2308756"
server_key_version: "encryptionkey@0"
I20260620 04:34:45.955933 23176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/wal/instance:
uuid: "c3a065980def4205908e29c20735648c"
format_stamp: "Formatted at 2026-06-20 04:34:45 on dist-test-slave-rzqm"
server_key: "f409e00e38d1238a52f4816859994a9c"
server_key_iv: "904c73ebf1dd4a395b237664f2308756"
server_key_version: "encryptionkey@0"
I20260620 04:34:45.959707 23176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 04:34:45.962123 23192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 04:34:45.963420 23176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 04:34:45.963531 23176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/wal
uuid: "c3a065980def4205908e29c20735648c"
format_stamp: "Formatted at 2026-06-20 04:34:45 on dist-test-slave-rzqm"
server_key: "f409e00e38d1238a52f4816859994a9c"
server_key_iv: "904c73ebf1dd4a395b237664f2308756"
server_key_version: "encryptionkey@0"
I20260620 04:34:45.963622 23176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 04:34:45.974030 23176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 04:34:45.977211 23176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 04:34:45.977372 23176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 04:34:45.984058 23176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35165
I20260620 04:34:45.984314 23244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35165 every 8 connection(s)
I20260620 04:34:45.985076 23176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/data/info.pb
I20260620 04:34:45.987943 23245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 04:34:45.990084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu as pid 23176
I20260620 04:34:45.990221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYF3Tc0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781929807654528-392-0/minicluster-data/master-2/wal/instance
I20260620 04:34:45.990559 392 external_mini_cluster.cc:1428] Setting key de23ca2412fb09a078deab4273b360b6
I20260620 04:34:45.992136 23245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:34:45 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 04:34:45 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:34:46.004022 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.993984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.0.98.62:41965)
0620 04:34:45.994112 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:45.994116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:45.994355 (+ 239us) server_negotiation.cc:407] Connection header received
0620 04:34:45.994494 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:45.994497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:45.994551 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:45.994615 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:45.995618 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.996127 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.997525 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.997748 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.997837 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.000376 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.000379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.000381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.000408 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.001912 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.002392 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.002396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.002398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.002471 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.002787 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.002790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.002792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.003039 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.003133 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.003717 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.003849 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 04:34:46.004985 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.993880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41965 (local address 127.0.0.1:36678)
0620 04:34:45.994233 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:45.994275 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:45.994395 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:45.994675 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:45.994684 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:45.995053 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:45.995481 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:45.995492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.996234 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.996237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:45.997408 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:45.997418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.997885 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.997889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:45.998011 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:45.998553 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:45.998569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:46.000239 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:46.002016 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:46.002025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:46.002037 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:46.002301 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:46.002549 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:46.002552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:46.002554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:46.002704 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:46.003173 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:46.003179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:46.003447 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 04:34:46.003615 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 04:34:46.003942 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":20,"spinlock_wait_cycles":8064,"thread_start_us":106,"threads_started":1}
I20260620 04:34:46.004985 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:45.992038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.0.98.60:35165)
0620 04:34:45.992542 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:45.992559 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:45.992596 (+ 37us) server_negotiation.cc:407] Connection header received
0620 04:34:45.993325 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:45.993346 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:45.993834 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:45.995021 (+ 1187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:45.995354 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.996242 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.997448 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:45.997852 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:45.998016 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.000630 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.000635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.000647 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.000668 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.002212 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.002604 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.002608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.002610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.002712 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.002965 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.002968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.002970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.003447 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.003642 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.003865 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.003979 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":349,"spinlock_wait_cycles":19968,"thread_start_us":169,"threads_started":1}
I20260620 04:34:46.006865 23245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 04:34:46 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 04:34:46 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 04:34:46.015313 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.007380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38985 (local address 127.0.0.1:42028)
0620 04:34:46.007478 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:46.007491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:46.007566 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:46.007809 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:46.007812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:46.007981 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:46.008077 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:46.008082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.008866 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.008869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:46.009442 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:46.009451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.009892 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.009896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:46.010005 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.010461 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:46.010473 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:46.011687 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:46.013828 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:46.013830 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:46.013832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:46.014022 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:46.014313 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:46.014317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:46.014319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:46.014408 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:46.014863 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:46.014866 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:46.014961 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 04:34:46.015037 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 04:34:46.015145 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 04:34:46.015400 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.007482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.0.98.61:38985)
0620 04:34:46.007608 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.007612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.007625 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:34:46.007664 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.007666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.007712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.007782 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.008193 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.008770 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.009544 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.009796 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.009885 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.011836 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.011841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.011843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.011875 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.013643 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.014115 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.014118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.014120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.014227 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.014526 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.014530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.014532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.014776 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.014854 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.015109 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.015246 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 04:34:46.016134 23245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } has no permanent_uuid. Determining permanent_uuid...
I20260620 04:34:46.016270 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.007562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.0.98.62:41965)
0620 04:34:46.007660 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.007664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.007676 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:34:46.007786 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.007789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.007829 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.007888 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.008470 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.008972 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.009717 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.009931 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.010013 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.012404 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.012408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.012411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.012437 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.014041 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.014526 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.014530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.014532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.014627 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.014893 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.014895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.014897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.015089 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.015378 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.016020 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.016132 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 04:34:46 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 04:34:46 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 04:34:46.023557 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.016646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47778)
0620 04:34:46.016731 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:46.016743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:46.016803 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:46.017125 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:46.017128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:46.017295 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:46.017380 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:46.017387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.018059 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.018062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:46.018565 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:46.018571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.018983 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.018986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:46.019067 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.019458 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:46.019471 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:46.020595 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:46.022195 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:46.022198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:46.022200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:46.022432 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:46.022689 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:46.022691 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:46.022693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:46.022789 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:46.023112 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:46.023114 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:46.023193 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 04:34:46.023284 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 04:34:46.023400 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 04:34:46.024034 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.016773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47778 (local address 127.0.98.60:35165)
0620 04:34:46.016955 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.016958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.016967 (+ 9us) server_negotiation.cc:407] Connection header received
0620 04:34:46.016991 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.016993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.017050 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.017115 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.017465 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.017972 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.018655 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.018895 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.018975 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.020703 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.020706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.020709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.020730 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.022093 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.022537 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.022539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.022540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.022608 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.022863 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.022867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.022868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.023033 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.023112 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.023763 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.023875 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":122}
I20260620 04:34:46.024847 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.017352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.0.98.61:38985)
0620 04:34:46.017450 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.017453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.017465 (+ 12us) server_negotiation.cc:407] Connection header received
0620 04:34:46.017501 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.017504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.017540 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.017602 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.018133 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.018616 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.019312 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.019535 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.019615 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.021985 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.021988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.021990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.022012 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.023343 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.023769 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.023773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.023774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.023853 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.024094 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.024097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.024099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.024278 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.024471 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.024599 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.024698 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 04:34:46.027016 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } attempt: 3
I20260620 04:34:46.027854 23245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c: Bootstrap starting.
I20260620 04:34:46.028333 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 04:34:46.030826 23245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:46.029532 23068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "af7b253ed87f442a9d99cec4674a800f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36688
I20260620 04:34:46.030987 23169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } attempt: 3
Jun 20 04:34:46 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:34:46.031777 23245 log.cc:824] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:46.034274 23245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c: No bootstrap required, opened a new log
I20260620 04:34:46.035405 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.027399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47780)
0620 04:34:46.027502 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:46.027516 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:46.027592 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:46.027919 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:46.027923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:46.028097 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:46.028197 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:46.028204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.028931 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.028935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:46.029511 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:46.029520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.029964 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.029967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:46.030049 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.030520 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:46.030534 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:46.031842 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:46.033818 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:46.033821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:46.033824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:46.034070 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:46.034352 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:46.034355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:46.034357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:46.034448 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:46.034882 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:46.034886 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:46.034997 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 04:34:46.035102 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 04:34:46.035241 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 04:34:46.035425 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.027602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.0.98.60:35165)
0620 04:34:46.027725 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.027728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.027738 (+ 10us) server_negotiation.cc:407] Connection header received
0620 04:34:46.027764 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.027767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.027807 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.027881 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.028308 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.028831 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.029619 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.029864 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.029954 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.031987 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.031990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.031992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.032021 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.033713 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.034162 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.034166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.034168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.034254 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.034531 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.034534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.034536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.034788 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.034885 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.035149 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.035276 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 04:34:46 dist-test-slave-rzqm krb5kdc[23013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781930085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 04:34:46.038353 23245 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } }
I20260620 04:34:46.038596 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f: Bootstrap starting.
I20260620 04:34:46.038630 23245 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a065980def4205908e29c20735648c, State: Initialized, Role: FOLLOWER
I20260620 04:34:46.038949 23245 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } }
I20260620 04:34:46.040272 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.031376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35165 (local address 127.0.0.1:47782)
0620 04:34:46.031544 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 04:34:46.031558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 04:34:46.031631 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 04:34:46.032281 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 04:34:46.032284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 04:34:46.032465 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 04:34:46.032579 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:46.032587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.033693 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.033697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:46.034347 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 04:34:46.034357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.034968 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.034972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 04:34:46.035052 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.035491 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 04:34:46.035505 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 04:34:46.036836 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 04:34:46.038823 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:46.038826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:46.038828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:46.039018 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:46.039286 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 04:34:46.039289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 04:34:46.039291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 04:34:46.039367 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 04:34:46.039786 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 04:34:46.039789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 04:34:46.039879 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 04:34:46.039963 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 04:34:46.040100 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 04:34:46.040288 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.031451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.0.98.60:35165)
0620 04:34:46.031921 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.031930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.031943 (+ 13us) server_negotiation.cc:407] Connection header received
0620 04:34:46.031997 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.032000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.032070 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.032687 (+ 617us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.032815 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.033577 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.034472 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.034845 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.034961 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.036960 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.036963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.036966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.036989 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.038714 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.039106 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.039109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.039111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.039196 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.039448 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.039451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.039453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.039657 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.039784 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.040024 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.040124 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":360,"thread_start_us":117,"threads_started":1}
I20260620 04:34:46.040268 23245 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:46.041445 23094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:46.042342 23094 log.cc:824] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:46.042840 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } } }
I20260620 04:34:46.042976 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:46.043391 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f: Bootstrap starting.
I20260620 04:34:46.044293 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f: No bootstrap required, opened a new log
W20260620 04:34:46.045502 23263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:46.045611 23263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:46.045996 23169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f: Neither blocks nor log segments found. Creating new log.
I20260620 04:34:46.046670 23169 log.cc:824] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f: Log is configured to *not* fsync() on all Append() calls
I20260620 04:34:46.047308 23094 raft_consensus.cc:348] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } }
I20260620 04:34:46.047536 23094 raft_consensus.cc:729] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7b253ed87f442a9d99cec4674a800f, State: Initialized, Role: FOLLOWER
I20260620 04:34:46.047883 23094 consensus_queue.cc:260] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } }
I20260620 04:34:46.049228 23094 sys_catalog.cc:564] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:46.048699 23176 master_runner.cc:186] Error getting master registration for 127.0.98.62:41965: OK, instance_id { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" instance_seqno: 1781930085565088 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:34:46.051409 23176 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 04:34:46.051543 23068 raft_consensus.cc:482] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 04:34:46.051725 23068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:46.052480 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f: No bootstrap required, opened a new log
I20260620 04:34:46.053320 23068 raft_consensus.cc:504] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } }
I20260620 04:34:46.054281 23068 leader_election.cc:288] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [CANDIDATE]: Term 1 election: Requested vote from peers dfa8dc6275114840b5f1b9545f26259f (127.0.98.61:38985), c3a065980def4205908e29c20735648c (127.0.98.60:35165)
I20260620 04:34:46.055337 23219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af7b253ed87f442a9d99cec4674a800f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c3a065980def4205908e29c20735648c"
I20260620 04:34:46.055554 23219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:46.056716 23219 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7b253ed87f442a9d99cec4674a800f in term 1.
I20260620 04:34:46.057623 23043 leader_election.cc:302] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af7b253ed87f442a9d99cec4674a800f, c3a065980def4205908e29c20735648c; no voters:
I20260620 04:34:46.058171 23284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 04:34:46.058576 23284 raft_consensus.cc:686] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [term 1 LEADER]: Becoming Leader. State: Replica: af7b253ed87f442a9d99cec4674a800f, State: Running, Role: LEADER
I20260620 04:34:46.058689 23169 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } }
I20260620 04:34:46.058935 23169 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa8dc6275114840b5f1b9545f26259f, State: Initialized, Role: FOLLOWER
I20260620 04:34:46.059290 23169 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } }
I20260620 04:34:46.059314 23284 consensus_queue.cc:237] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } }
I20260620 04:34:46.060741 23169 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 04:34:46.060786 23142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af7b253ed87f442a9d99cec4674a800f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dfa8dc6275114840b5f1b9545f26259f"
I20260620 04:34:46.060984 23142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [term 0 FOLLOWER]: Advancing to term 1
I20260620 04:34:46.062397 23142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7b253ed87f442a9d99cec4674a800f in term 1.
I20260620 04:34:46.058871 23026 master_runner.cc:186] Error getting master registration for 127.0.98.61:38985: OK, instance_id { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" instance_seqno: 1781930085776718 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 04:34:46.062611 23026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 04:34:46.062755 23047 tablet.cc:2369] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 04:34:46.063448 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } } }
I20260620 04:34:46.064462 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:46.065056 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [sys.catalog]: SysCatalogTable state changed. Reason: New leader af7b253ed87f442a9d99cec4674a800f. Latest consensus state: current_term: 1 leader_uuid: "af7b253ed87f442a9d99cec4674a800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } } }
I20260620 04:34:46.065256 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [sys.catalog]: This master's current role is: LEADER
W20260620 04:34:46.068099 23278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:46.068173 23278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:46.068511 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af7b253ed87f442a9d99cec4674a800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } } }
I20260620 04:34:46.068621 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [sys.catalog]: This master's current role is: LEADER
W20260620 04:34:46.068773 23299 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 04:34:46.068882 23299 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 04:34:46.071244 23301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 04:34:46.072225 23301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 04:34:46.083822 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.057372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.0.98.62:41965)
0620 04:34:46.058059 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.058063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.063017 (+ 4954us) server_negotiation.cc:407] Connection header received
0620 04:34:46.063194 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.063197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.063245 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.063292 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.064249 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.065062 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.075916 (+ 10854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.076274 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.076350 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.078266 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.078270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.078273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.078302 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.079923 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.080403 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.080407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.080409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.080499 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.080818 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.080821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.080823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.081075 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.081181 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.081562 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.083630 (+ 2068us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609}
I20260620 04:34:46.085417 23219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [term 1 FOLLOWER]: Refusing update from remote peer af7b253ed87f442a9d99cec4674a800f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:46.086242 23285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:46.086982 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.061349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.0.98.61:38985)
0620 04:34:46.061461 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.061465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.067433 (+ 5968us) server_negotiation.cc:407] Connection header received
0620 04:34:46.067608 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.067611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.067658 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.067707 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.077572 (+ 9865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.078330 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.079470 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.079850 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.079935 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.081683 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.081687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.081690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.081717 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.083392 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.085610 (+ 2218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.085613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.085615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.085708 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.086004 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.086007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.086009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.086238 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.086364 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.086703 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.086821 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 04:34:46.090349 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:46.062029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.0.98.60:35165)
0620 04:34:46.063332 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.063336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.081972 (+ 18636us) server_negotiation.cc:407] Connection header received
0620 04:34:46.082206 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.082210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.082265 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.082315 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.083059 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.083779 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.084907 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:46.085287 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:46.085362 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:46.086932 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 04:34:46.086936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 04:34:46.086939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 04:34:46.086968 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 04:34:46.088549 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.089060 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.089063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.089065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.089158 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 04:34:46.089504 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 04:34:46.089507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 04:34:46.089508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 04:34:46.089733 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 04:34:46.089862 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:46.090072 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 04:34:46.090205 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1209}
I20260620 04:34:46.092180 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [sys.catalog]: SysCatalogTable state changed. Reason: New leader af7b253ed87f442a9d99cec4674a800f. Latest consensus state: current_term: 1 leader_uuid: "af7b253ed87f442a9d99cec4674a800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } } }
I20260620 04:34:46.092298 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:46.094238 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af7b253ed87f442a9d99cec4674a800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } } }
I20260620 04:34:46.094347 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:46.095579 23301 catalog_manager.cc:1347] Generated new cluster ID: 07be4edb06304498993904dd6cf52a7f
I20260620 04:34:46.095649 23301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 04:34:46.097806 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af7b253ed87f442a9d99cec4674a800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } } }
I20260620 04:34:46.097954 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:46.098784 23304 mvcc.cc:205] Tried to move back new op lower bound from 7298785632598454272 to 7298785632517955584. Current Snapshot: MvccSnapshot[applied={T|T < 7298785632598454272 or (T in {7298785632598454272})}]
I20260620 04:34:46.101815 23142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [term 1 FOLLOWER]: Refusing update from remote peer af7b253ed87f442a9d99cec4674a800f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 04:34:46.102778 23285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 04:34:46.110381 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [sys.catalog]: SysCatalogTable state changed. Reason: New leader af7b253ed87f442a9d99cec4674a800f. Latest consensus state: current_term: 1 leader_uuid: "af7b253ed87f442a9d99cec4674a800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } } }
I20260620 04:34:46.110535 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:46.110632 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af7b253ed87f442a9d99cec4674a800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } } }
I20260620 04:34:46.110723 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 04:34:46.117174 23305 mvcc.cc:205] Tried to move back new op lower bound from 7298785632598454272 to 7298785632517955584. Current Snapshot: MvccSnapshot[applied={T|T < 7298785632598454272 or (T in {7298785632598454272})}]
I20260620 04:34:46.133544 23309 mvcc.cc:205] Tried to move back new op lower bound from 7298785632598454272 to 7298785632517955584. Current Snapshot: MvccSnapshot[applied={T|T < 7298785632598454272}]
I20260620 04:34:46.160492 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af7b253ed87f442a9d99cec4674a800f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7b253ed87f442a9d99cec4674a800f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } peers { permanent_uuid: "dfa8dc6275114840b5f1b9545f26259f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38985 } } peers { permanent_uuid: "c3a065980def4205908e29c20735648c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35165 } } }
I20260620 04:34:46.160673 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f [sys.catalog]: This master's current role is: LEADER
I20260620 04:34:46.170696 23301 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 04:34:46.172039 23301 catalog_manager.cc:1504] Loading token signing keys...
I20260620 04:34:46.193675 23301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af7b253ed87f442a9d99cec4674a800f: Generated new TSK 0
I20260620 04:34:46.194545 23301 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 04:34:46.222404 23097 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.219003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.0.98.62:41965)
0620 04:34:46.219148 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.219153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.221189 (+ 2036us) server_negotiation.cc:407] Connection header received
0620 04:34:46.221417 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.221421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.221480 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.221537 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.222260 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36706: BlockingRecv error: recv got EOF from 127.0.0.1:36706 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 04:34:46.224455 23250 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.220516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.0.98.60:35165)
0620 04:34:46.220631 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.220635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.223149 (+ 2514us) server_negotiation.cc:407] Connection header received
0620 04:34:46.223425 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.223428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.223482 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.223534 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.224332 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47798: BlockingRecv error: recv got EOF from 127.0.0.1:47798 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 04:34:46.225265 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.219868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.0.98.61:38985)
0620 04:34:46.219983 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.219988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.222965 (+ 2977us) server_negotiation.cc:407] Connection header received
0620 04:34:46.223679 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.223683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.223749 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.223802 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.225148 (+ 1346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42048: BlockingRecv error: recv got EOF from 127.0.0.1:42048 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 04:34:46.230106 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.228485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.0.98.61:38985)
0620 04:34:46.228592 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.228595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.229093 (+ 498us) server_negotiation.cc:407] Connection header received
0620 04:34:46.229312 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.229316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.229391 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.229444 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.229998 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42050: BlockingRecv error: recv got EOF from 127.0.0.1:42050 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 04:34:46.235088 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.233715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.0.98.61:38985)
0620 04:34:46.233812 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.233816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.234267 (+ 451us) server_negotiation.cc:407] Connection header received
0620 04:34:46.234429 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.234432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.234479 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.234525 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.234987 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42058: BlockingRecv error: recv got EOF from 127.0.0.1:42058 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 04:34:46.240576 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.239151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.0.98.61:38985)
0620 04:34:46.239247 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.239251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.239720 (+ 469us) server_negotiation.cc:407] Connection header received
0620 04:34:46.239898 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.239901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.239948 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.239994 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.240467 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42062: BlockingRecv error: recv got EOF from 127.0.0.1:42062 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 04:34:46.248788 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.247021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.0.98.61:38985)
0620 04:34:46.247174 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.247179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.247740 (+ 561us) server_negotiation.cc:407] Connection header received
0620 04:34:46.247957 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.247961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.248021 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.248078 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.248660 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42078: BlockingRecv error: recv got EOF from 127.0.0.1:42078 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 04:34:46.260291 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.259148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.0.98.61:38985)
0620 04:34:46.259271 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.259275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.259481 (+ 206us) server_negotiation.cc:407] Connection header received
0620 04:34:46.259620 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.259623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.259671 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.259738 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.260164 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42084: BlockingRecv error: recv got EOF from 127.0.0.1:42084 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 04:34:46.279495 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.278288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42088 (local address 127.0.98.61:38985)
0620 04:34:46.278412 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.278417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.278648 (+ 231us) server_negotiation.cc:407] Connection header received
0620 04:34:46.278788 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.278791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.278839 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.278910 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.279361 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42088: BlockingRecv error: recv got EOF from 127.0.0.1:42088 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 04:34:46.314639 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.313420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.0.98.61:38985)
0620 04:34:46.313559 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.313563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.313674 (+ 111us) server_negotiation.cc:407] Connection header received
0620 04:34:46.313821 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.313824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.313878 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.313962 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.314499 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42090: BlockingRecv error: recv got EOF from 127.0.0.1:42090 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 04:34:46.381636 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.380541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.0.98.61:38985)
0620 04:34:46.380670 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.380673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.380797 (+ 124us) server_negotiation.cc:407] Connection header received
0620 04:34:46.380886 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.380888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.380938 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.381068 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.381533 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42098: BlockingRecv error: recv got EOF from 127.0.0.1:42098 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 04:34:46.512907 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.511597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.0.98.61:38985)
0620 04:34:46.511750 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.511755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.512043 (+ 288us) server_negotiation.cc:407] Connection header received
0620 04:34:46.512189 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.512192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.512243 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.512317 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.512800 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42102: BlockingRecv error: recv got EOF from 127.0.0.1:42102 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 04:34:46.772073 23175 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 04:34:46.771055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.0.98.61:38985)
0620 04:34:46.771213 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:46.771217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:46.771312 (+ 95us) server_negotiation.cc:407] Connection header received
0620 04:34:46.771395 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:46.771398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:46.771449 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:46.771532 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 04:34:46.771976 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42114: BlockingRecv error: recv got EOF from 127.0.0.1:42114 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 04:34:47.046900 23263 catalog_manager.cc:1259] Loaded cluster ID: 07be4edb06304498993904dd6cf52a7f
I20260620 04:34:47.046989 23263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c: loading cluster ID for follower catalog manager: success
I20260620 04:34:47.048772 23263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c: acquiring CA information for follower catalog manager: success
I20260620 04:34:47.050118 23263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3a065980def4205908e29c20735648c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:34:47.070186 23299 catalog_manager.cc:1259] Loaded cluster ID: 07be4edb06304498993904dd6cf52a7f
I20260620 04:34:47.070277 23299 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f: loading cluster ID for follower catalog manager: success
I20260620 04:34:47.071993 23299 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f: acquiring CA information for follower catalog manager: success
I20260620 04:34:47.073225 23299 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dfa8dc6275114840b5f1b9545f26259f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 04:34:47.290107 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:47.286142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.0.98.61:38985)
0620 04:34:47.286351 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:47.286355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:47.286481 (+ 126us) server_negotiation.cc:407] Connection header received
0620 04:34:47.286611 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:47.286614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:47.286746 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:47.286845 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:47.287167 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:47.287708 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:47.288643 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:47.288883 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:47.288964 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:47.289302 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:47.289748 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:47.289842 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:47.289924 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 04:34:47.289978 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":58,"threads_started":1}
I20260620 04:34:47.296716 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:47.293149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.0.98.60:35165)
0620 04:34:47.293368 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:47.293372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:47.293388 (+ 16us) server_negotiation.cc:407] Connection header received
0620 04:34:47.293527 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:47.293530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:47.293642 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:47.293736 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:47.293894 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:47.294390 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:47.295230 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:47.295512 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:47.295602 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:47.295883 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:47.296358 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:47.296450 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:47.296542 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 04:34:47.296593 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
I20260620 04:34:47.303758 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:47.299532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.0.98.61:38985)
0620 04:34:47.299741 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:47.299745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:47.300239 (+ 494us) server_negotiation.cc:407] Connection header received
0620 04:34:47.300373 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:47.300377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:47.300430 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:47.300473 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:47.300795 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:47.301559 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:47.302491 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:47.302814 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:47.302911 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:47.303272 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:47.303360 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:47.303423 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:47.303559 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 04:34:47.303615 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 04:34:47.304280 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:47.299919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.0.98.60:35165)
0620 04:34:47.300029 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:47.300033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:47.300362 (+ 329us) server_negotiation.cc:407] Connection header received
0620 04:34:47.300406 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:47.300409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:47.300454 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:47.300516 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:47.300718 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:47.301272 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:47.303041 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:47.303279 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:47.303381 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:47.303658 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:47.303744 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:47.303820 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:47.304112 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 04:34:47.304166 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 04:34:47.304785 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 04:34:47.299501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36708 (local address 127.0.98.62:41965)
0620 04:34:47.299876 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 04:34:47.299881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 04:34:47.299895 (+ 14us) server_negotiation.cc:407] Connection header received
0620 04:34:47.299970 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 04:34:47.299973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 04:34:47.300128 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 04:34:47.300244 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 04:34:47.300510 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:47.301372 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:47.302533 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 04:34:47.302879 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 04:34:47.302965 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 04:34:47.303547 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 04:34:47.303903 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 04:34:47.303993 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 04:34:47.304554 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 04:34:47.304619 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":69,"threads_started":1}
I20260620 04:34:47.311354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 23026
I20260620 04:34:47.316761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 23101
I20260620 04:34:47.321802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYF3Tc0/build/debug/bin/kudu with pid 23176
2026-06-20T04:34:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3033 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5994 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279658 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST